[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:23 dist-test-slave-d5wc krb5kdc[22072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:23 dist-test-slave-d5wc krb5kdc[22072](info): set up 2 sockets
Sep 28 21:06:23 dist-test-slave-d5wc krb5kdc[22072](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 21:06:24.564283 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:24 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:24.697782 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37741
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:24.813642 22085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:24.813977 22085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:24.814177 22085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:24.818462 22085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:24.818555 22085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:24.818589 22085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:24.818627 22085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:24.818657 22085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:24.824720 22085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:24.825855 22085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:24.826972 22085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:24.836148 22090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:24.836853 22091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:24.836853 22085 server_base.cc:1034] running on GCE node
W20260928 21:06:24.837194 22093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:24.837805 22085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:24.838994 22085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:24.840193 22085 hybrid_clock.cc:648] HybridClock initialized: now 1790629584840157 us; error 48 us; skew 500 ppm
Sep 28 21:06:24 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629584, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:24.843065 22085 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:24.844118 22085 webserver.cc:460] Webserver started at http://127.21.140.126:45957/ using document root <none> and password file <none>
I20260928 21:06:24.844715 22085 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:24.844820 22085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:24.845053 22085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:24.847213 22085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "925daf669017462497fe9e2afd7c70b4"
format_stamp: "Formatted at 2026-09-28 21:06:24 on dist-test-slave-d5wc"
server_key: "f103bbc5e4f7c974e980d85be882fed0"
server_key_iv: "3de3d8f952f4b77526ca1aaf2b9710f6"
server_key_version: "encryptionkey@0"
I20260928 21:06:24.847806 22085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "925daf669017462497fe9e2afd7c70b4"
format_stamp: "Formatted at 2026-09-28 21:06:24 on dist-test-slave-d5wc"
server_key: "f103bbc5e4f7c974e980d85be882fed0"
server_key_iv: "3de3d8f952f4b77526ca1aaf2b9710f6"
server_key_version: "encryptionkey@0"
I20260928 21:06:24.851598 22085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 21:06:24.854221 22100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:24.855707 22085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:06:24.855873 22085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "925daf669017462497fe9e2afd7c70b4"
format_stamp: "Formatted at 2026-09-28 21:06:24 on dist-test-slave-d5wc"
server_key: "f103bbc5e4f7c974e980d85be882fed0"
server_key_iv: "3de3d8f952f4b77526ca1aaf2b9710f6"
server_key_version: "encryptionkey@0"
I20260928 21:06:24.856072 22085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:24.880375 22085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:24.884040 22085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:24.884307 22085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:24.892132 22085 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37741
I20260928 21:06:24.892177 22152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37741 every 8 connection(s)
I20260928 21:06:24.893148 22085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:24.894861 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22085
I20260928 21:06:24.895037 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:24.896185 22153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:24.896420 22065 external_mini_cluster.cc:1428] Setting key db2991efcedde35ec3aaf271c2a8d4fa
I20260928 21:06:24.902652 22153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4: Bootstrap starting.
I20260928 21:06:24.904830 22153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:24.905561 22153 log.cc:824] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:24.907557 22153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4: No bootstrap required, opened a new log
I20260928 21:06:24.911228 22153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925daf669017462497fe9e2afd7c70b4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37741 } }
I20260928 21:06:24.911499 22153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:24.911582 22153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925daf669017462497fe9e2afd7c70b4, State: Initialized, Role: FOLLOWER
I20260928 21:06:24.912117 22153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925daf669017462497fe9e2afd7c70b4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37741 } }
I20260928 21:06:24.912302 22153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:24.912381 22153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:24.912484 22153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:24.913522 22153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925daf669017462497fe9e2afd7c70b4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37741 } }
I20260928 21:06:24.913941 22153 leader_election.cc:302] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 925daf669017462497fe9e2afd7c70b4; no voters:
I20260928 21:06:24.914402 22153 leader_election.cc:288] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:24.914491 22158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 21:06:24 dist-test-slave-d5wc krb5kdc[22072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:24.914740 22158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [term 1 LEADER]: Becoming Leader. State: Replica: 925daf669017462497fe9e2afd7c70b4, State: Running, Role: LEADER
I20260928 21:06:24.915074 22158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925daf669017462497fe9e2afd7c70b4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37741 } }
I20260928 21:06:24.915802 22153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:24.916908 22159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "925daf669017462497fe9e2afd7c70b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925daf669017462497fe9e2afd7c70b4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37741 } } }
I20260928 21:06:24.917054 22159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:24.916981 22160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 925daf669017462497fe9e2afd7c70b4. Latest consensus state: current_term: 1 leader_uuid: "925daf669017462497fe9e2afd7c70b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925daf669017462497fe9e2afd7c70b4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37741 } } }
I20260928 21:06:24.919131 22160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4 [sys.catalog]: This master's current role is: LEADER
W20260928 21:06:24.921741 22173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:06:24.921841 22173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:06:24.921991 22167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:24.922441 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:24.899772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.21.140.126:37741)
0928 21:06:24.900202 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:24.900218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:24.901309 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 21:06:24.904505 (+ 3196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:24.904521 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:24.904846 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:24.905226 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:24.907609 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:24.908787 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:24.910173 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:24.910668 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:24.910864 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:24.915705 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:24.915712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:24.915728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:24.915766 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:24.918172 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:24.919014 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:24.919024 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:24.919032 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:24.919197 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:24.919595 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:24.919598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:24.919600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:24.920019 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:24.920269 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:24.920858 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 21:06:24.921151 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
I20260928 21:06:24.922768 22167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:24.928532 22167 catalog_manager.cc:1347] Generated new cluster ID: f9c995bf4ac44d05ab71f82b1e995747
I20260928 21:06:24.928649 22167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:24.952888 22167 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:24.954281 22167 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:24.964733 22167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 925daf669017462497fe9e2afd7c70b4: Generated new TSK 0
I20260928 21:06:24.965598 22167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:25.002120 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37741
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:25.121152 22179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:25.121460 22179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:25.121510 22179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:25.125728 22179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:25.125808 22179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:25.125936 22179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:25.131639 22179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:25.132824 22179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:25.133891 22179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:25.144330 22185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:25.144330 22184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:25.144369 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:25.144626 22179 server_base.cc:1034] running on GCE node
I20260928 21:06:25.145303 22179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:25.146076 22179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:25.147255 22179 hybrid_clock.cc:648] HybridClock initialized: now 1790629585147237 us; error 32 us; skew 500 ppm
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:25.150271 22179 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:25.151338 22179 webserver.cc:460] Webserver started at http://127.21.140.65:44363/ using document root <none> and password file <none>
I20260928 21:06:25.151932 22179 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:25.152027 22179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:25.152278 22179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:25.154417 22179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "e5295a25eccc480aa747e22b33b0a56c"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "1813412da3ac534ae13c63f878e48f51"
server_key_iv: "5b35c8ddb59894a80d8c3f2f76503c1e"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.155001 22179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "e5295a25eccc480aa747e22b33b0a56c"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "1813412da3ac534ae13c63f878e48f51"
server_key_iv: "5b35c8ddb59894a80d8c3f2f76503c1e"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.159044 22179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:06:25.161813 22194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.163089 22179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:06:25.163280 22179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "e5295a25eccc480aa747e22b33b0a56c"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "1813412da3ac534ae13c63f878e48f51"
server_key_iv: "5b35c8ddb59894a80d8c3f2f76503c1e"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.163407 22179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:25.193941 22179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:25.197060 22179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:25.197291 22179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:25.197906 22179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:25.198779 22179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:25.198858 22179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.198926 22179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:25.198973 22179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.208338 22179 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:36067
I20260928 21:06:25.208384 22307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:36067 every 8 connection(s)
I20260928 21:06:25.209353 22179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:25.219606 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22179
I20260928 21:06:25.219703 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:25.220059 22065 external_mini_cluster.cc:1428] Setting key 32396b0789867960cb1649d252cea57b
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:06:25.234110 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.211311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:35301 (local address 127.21.140.126:37741)
0928 21:06:25.211486 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:25.211491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:25.212582 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 21:06:25.213412 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:25.213414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:25.213467 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:25.213544 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:25.214857 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.215457 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.216802 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.217091 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.217232 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.229548 (+ 12316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:25.229552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:25.229554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:25.229583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:25.231337 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.231934 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.231938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.231940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.232055 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.232508 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.232515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.232519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.232798 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:25.232880 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:25.233694 (+ 814us) server_negotiation.cc:299] Negotiation successful
0928 21:06:25.233862 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 21:06:25.235002 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.211739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37741 (local address 127.21.140.65:35301)
0928 21:06:25.212423 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:25.212465 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:25.213225 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:25.213711 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:25.213718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:25.214209 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:25.214696 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:25.214710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.215585 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.215588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:25.216685 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:25.216695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.217274 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.217277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:25.217433 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.227293 (+ 9860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:25.227320 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:25.229307 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:25.231476 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:25.231483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:25.231502 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:25.231814 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:25.232141 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:25.232144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:25.232146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:25.232357 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:25.232970 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:25.232979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:25.233355 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 21:06:25.233591 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 21:06:25.233954 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":112,"threads_started":1}
I20260928 21:06:25.236428 22308 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37741
I20260928 21:06:25.236722 22308 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:25.237391 22308 heartbeater.cc:511] Master 127.21.140.126:37741 requested a full tablet report, sending...
I20260928 21:06:25.239136 22117 ts_manager.cc:194] Registered new tserver with Master: e5295a25eccc480aa747e22b33b0a56c (127.21.140.65:36067)
I20260928 21:06:25.240295 22117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:35301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:25.249025 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37741
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:25.361814 22313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:25.362196 22313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:25.362288 22313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:25.367436 22313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:25.367583 22313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:25.367715 22313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:25.373410 22313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:25.374639 22313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:25.375702 22313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:06:25.388332 22313 server_base.cc:1034] running on GCE node
W20260928 21:06:25.388372 22318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:25.388348 22319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:25.388326 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:25.388947 22313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:25.389654 22313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:25.390899 22313 hybrid_clock.cc:648] HybridClock initialized: now 1790629585390874 us; error 43 us; skew 500 ppm
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:25.393944 22313 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:25.395028 22313 webserver.cc:460] Webserver started at http://127.21.140.66:43887/ using document root <none> and password file <none>
I20260928 21:06:25.395614 22313 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:25.395712 22313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:25.395946 22313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:25.398087 22313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "1058d284e98a47f9a5a6cc19dc3f4cfb"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "cb6ad2ee2043d8f0cd1bc6c13a1de146"
server_key_iv: "4b05f613d3edda9bccbd470ae26f2d7b"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.398655 22313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "1058d284e98a47f9a5a6cc19dc3f4cfb"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "cb6ad2ee2043d8f0cd1bc6c13a1de146"
server_key_iv: "4b05f613d3edda9bccbd470ae26f2d7b"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.402417 22313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:25.404879 22328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.406303 22313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:25.406435 22313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "1058d284e98a47f9a5a6cc19dc3f4cfb"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "cb6ad2ee2043d8f0cd1bc6c13a1de146"
server_key_iv: "4b05f613d3edda9bccbd470ae26f2d7b"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.406565 22313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:25.424347 22313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:25.427754 22313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:25.427985 22313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:25.428807 22313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:25.429723 22313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:25.429811 22313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.429905 22313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:25.429956 22313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.439432 22313 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:43245
I20260928 21:06:25.439468 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:43245 every 8 connection(s)
I20260928 21:06:25.440341 22313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:25.445844 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22313
I20260928 21:06:25.446018 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:25.446271 22065 external_mini_cluster.cc:1428] Setting key e140f8c40a69f2dae731eceb1037cb6c
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:25.456344 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.442376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:41541 (local address 127.21.140.126:37741)
0928 21:06:25.442527 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:25.442532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:25.443847 (+ 1315us) server_negotiation.cc:407] Connection header received
0928 21:06:25.444731 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:25.444734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:25.444783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:25.445001 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:25.446360 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.446951 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.448496 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.448937 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.449067 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.452259 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:25.452266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:25.452273 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:25.452308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:25.454104 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.454629 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.454635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.454639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.454735 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.455064 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.455067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.455069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.455336 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:25.455448 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:25.455985 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 21:06:25.456137 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:25.457490 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.442802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37741 (local address 127.21.140.66:41541)
0928 21:06:25.443676 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:25.443722 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:25.444489 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:25.445160 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:25.445167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:25.445519 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:25.446202 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:25.446220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.447094 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.447098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:25.448351 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:25.448361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.449087 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.449090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:25.449229 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.449923 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:25.449948 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:25.452059 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:25.454242 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:25.454247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:25.454261 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:25.454520 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:25.454832 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:25.454834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:25.454835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:25.454966 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:25.455440 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:25.455445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:25.455683 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 21:06:25.455896 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 21:06:25.456353 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":531,"threads_started":1}
I20260928 21:06:25.458694 22442 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37741
I20260928 21:06:25.458937 22442 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:25.459467 22442 heartbeater.cc:511] Master 127.21.140.126:37741 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:06:25.460752 22117 ts_manager.cc:194] Registered new tserver with Master: 1058d284e98a47f9a5a6cc19dc3f4cfb (127.21.140.66:43245)
I20260928 21:06:25.461616 22117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:41541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:25.477761 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37741
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:25.591296 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:25.591636 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:25.591722 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:25.595875 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:25.595960 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:25.596056 22447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:25.601399 22447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:25.602546 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:25.603571 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:25.613523 22453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:25.613516 22452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:25.613584 22455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:25.613703 22447 server_base.cc:1034] running on GCE node
I20260928 21:06:25.614267 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:25.614934 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:25.616138 22447 hybrid_clock.cc:648] HybridClock initialized: now 1790629585616112 us; error 62 us; skew 500 ppm
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:25.619321 22447 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:25.620307 22447 webserver.cc:460] Webserver started at http://127.21.140.67:40509/ using document root <none> and password file <none>
I20260928 21:06:25.620904 22447 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:25.621014 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:25.621371 22447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:25.623749 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "4eb63bccfe5c4116a1a413ebd0f3055f"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "f9e77c57ef994b57b952f778265dfe8d"
server_key_iv: "7e058d6b7f4f0f07d64b94056cb54ff8"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.624389 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "4eb63bccfe5c4116a1a413ebd0f3055f"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "f9e77c57ef994b57b952f778265dfe8d"
server_key_iv: "7e058d6b7f4f0f07d64b94056cb54ff8"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.628391 22447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:25.630913 22462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.632251 22447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:25.632390 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "4eb63bccfe5c4116a1a413ebd0f3055f"
format_stamp: "Formatted at 2026-09-28 21:06:25 on dist-test-slave-d5wc"
server_key: "f9e77c57ef994b57b952f778265dfe8d"
server_key_iv: "7e058d6b7f4f0f07d64b94056cb54ff8"
server_key_version: "encryptionkey@0"
I20260928 21:06:25.632511 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:25.644987 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:25.648270 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:25.648486 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:25.649091 22447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:25.649906 22447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:25.650041 22447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.650126 22447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:25.650180 22447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:25.659838 22447 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:38453
I20260928 21:06:25.659864 22575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:38453 every 8 connection(s)
I20260928 21:06:25.660791 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:25.664383 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22447
I20260928 21:06:25.664502 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:25.664790 22065 external_mini_cluster.cc:1428] Setting key d3cd567dc5b3617d9378dd520c77d4a7
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:25.675877 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.662775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:41923 (local address 127.21.140.126:37741)
0928 21:06:25.662921 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:25.662925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:25.664359 (+ 1434us) server_negotiation.cc:407] Connection header received
0928 21:06:25.665511 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:25.665514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:25.665562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:25.665663 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:25.667218 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.667888 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.669124 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.669337 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.669421 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.672277 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:25.672280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:25.672282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:25.672308 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:25.673847 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.674407 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.674409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.674411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.674484 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.674827 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.674830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.674831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.675025 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:25.675112 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:25.675637 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 21:06:25.675752 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:06:25.676723 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.663195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37741 (local address 127.21.140.67:41923)
0928 21:06:25.664149 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:25.664212 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:25.665260 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:25.665849 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:25.665859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:25.666421 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:25.666980 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:25.667023 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.668020 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.668023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:25.668995 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:25.669005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.669473 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.669476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:25.669623 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.670383 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:25.670410 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:25.672083 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:25.673956 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:25.673960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:25.674030 (+ 70us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:25.674318 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:25.674565 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:25.674567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:25.674568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:25.674699 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:25.675119 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:25.675123 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:25.675378 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 21:06:25.675532 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 21:06:25.675801 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":553,"threads_started":1}
I20260928 21:06:25.678025 22576 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37741
I20260928 21:06:25.678318 22576 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:25.678828 22576 heartbeater.cc:511] Master 127.21.140.126:37741 requested a full tablet report, sending...
I20260928 21:06:25.679811 22117 ts_manager.cc:194] Registered new tserver with Master: 4eb63bccfe5c4116a1a413ebd0f3055f (127.21.140.67:38453)
I20260928 21:06:25.680387 22117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:41923
I20260928 21:06:25.691326 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:25.718859 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.708466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.21.140.65:36067)
0928 21:06:25.708812 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:25.708825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:25.708850 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:06:25.708902 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:25.708907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:25.709094 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:25.709294 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:25.710044 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.710901 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.711770 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.712145 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.712237 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.714988 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:25.714998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:25.715013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:25.715038 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:25.716855 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.717328 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.717341 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.717342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.717424 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.717722 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.717726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.717728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.718065 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:25.718311 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:25.718500 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:06:25.718658 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
W20260928 21:06:25.719913 22222 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46836
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:25.746460 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:25.737720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.21.140.65:36067)
0928 21:06:25.737890 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:25.737895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:25.738026 (+ 131us) server_negotiation.cc:407] Connection header received
0928 21:06:25.738206 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:25.738208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:25.738253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:25.738325 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:25.739066 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.739593 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.740425 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:25.740683 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:25.740818 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:25.743445 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:25.743449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:25.743451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:25.743475 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:25.744987 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.745400 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.745403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.745404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.745478 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:25.745723 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:25.745725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:25.745726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:25.745888 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:25.746049 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:25.746236 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 21:06:25.746336 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:06:25.747612 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22179
I20260928 21:06:25.753590 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22313
I20260928 21:06:25.762027 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22447
I20260928 21:06:25.769114 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22085
2026-09-28T21:06:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2293 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22590](info): set up 2 sockets
Sep 28 21:06:25 dist-test-slave-d5wc krb5kdc[22590](info): commencing operation
krb5kdc: starting...
W20260928 21:06:26.823266 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:26 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:26.957640 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34859
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:27.071933 22603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:27.072240 22603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:27.072283 22603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:27.076726 22603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:27.076815 22603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:27.076835 22603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:27.076853 22603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:27.076869 22603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:27.082683 22603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:34859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:27.083911 22603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:27.084991 22603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:27.094058 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:27.094102 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:27.094082 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:27.094302 22603 server_base.cc:1034] running on GCE node
I20260928 21:06:27.095084 22603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:27.096431 22603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:27.097676 22603 hybrid_clock.cc:648] HybridClock initialized: now 1790629587097656 us; error 35 us; skew 500 ppm
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:27.100857 22603 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:27.101922 22603 webserver.cc:460] Webserver started at http://127.21.140.126:44235/ using document root <none> and password file <none>
I20260928 21:06:27.102878 22603 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:27.103071 22603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:27.103332 22603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:27.105577 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "34b6de2669774289b08102086db80cbe"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "983dbe3b8a251eaf740f863ad9206562"
server_key_iv: "97e0e81ce8318f5c021276629223594e"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.106249 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "34b6de2669774289b08102086db80cbe"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "983dbe3b8a251eaf740f863ad9206562"
server_key_iv: "97e0e81ce8318f5c021276629223594e"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.110460 22603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:06:27.113207 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.114759 22603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:27.114926 22603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "34b6de2669774289b08102086db80cbe"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "983dbe3b8a251eaf740f863ad9206562"
server_key_iv: "97e0e81ce8318f5c021276629223594e"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.115082 22603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:27.127238 22603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:27.130966 22603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:27.131220 22603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:27.138896 22603 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:34859
I20260928 21:06:27.138924 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:34859 every 8 connection(s)
I20260928 21:06:27.139955 22603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:27.143046 22671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:27.144910 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22603
I20260928 21:06:27.145066 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:27.145349 22065 external_mini_cluster.cc:1428] Setting key b2179411a00f34855e25ac10f30a4f48
I20260928 21:06:27.150440 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe: Bootstrap starting.
I20260928 21:06:27.153462 22671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:27.154572 22671 log.cc:824] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:27.157236 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe: No bootstrap required, opened a new log
I20260928 21:06:27.161284 22671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6de2669774289b08102086db80cbe" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34859 } }
I20260928 21:06:27.161597 22671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:27.161690 22671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b6de2669774289b08102086db80cbe, State: Initialized, Role: FOLLOWER
I20260928 21:06:27.162300 22671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6de2669774289b08102086db80cbe" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34859 } }
I20260928 21:06:27.162468 22671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:27.162560 22671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:27.162647 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.146966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.21.140.126:34859)
0928 21:06:27.147436 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.147454 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.147494 (+ 40us) server_negotiation.cc:407] Connection header received
0928 21:06:27.148212 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.148232 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.148615 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.148931 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:27.150026 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.151225 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.152504 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.153252 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.153403 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.156594 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:27.156601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:27.156624 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:27.156653 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:27.158847 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.159421 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.159433 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.159439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.159582 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.159941 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.159945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.159947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.160432 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:27.160738 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.161065 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.161401 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":161,"threads_started":1}
I20260928 21:06:27.162696 22671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:27.163817 22671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6de2669774289b08102086db80cbe" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34859 } }
I20260928 21:06:27.164197 22671 leader_election.cc:302] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b6de2669774289b08102086db80cbe; no voters:
I20260928 21:06:27.164572 22671 leader_election.cc:288] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:27.164732 22676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:27.164965 22676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [term 1 LEADER]: Becoming Leader. State: Replica: 34b6de2669774289b08102086db80cbe, State: Running, Role: LEADER
I20260928 21:06:27.165531 22676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6de2669774289b08102086db80cbe" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34859 } }
I20260928 21:06:27.166031 22671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:27.167305 22678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b6de2669774289b08102086db80cbe. Latest consensus state: current_term: 1 leader_uuid: "34b6de2669774289b08102086db80cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6de2669774289b08102086db80cbe" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34859 } } }
I20260928 21:06:27.167447 22678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:27.167757 22677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b6de2669774289b08102086db80cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6de2669774289b08102086db80cbe" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34859 } } }
I20260928 21:06:27.167843 22677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:27.167912 22686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:27.170992 22686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:27.176800 22686 catalog_manager.cc:1347] Generated new cluster ID: bcbe7d89abcf463da0f4a94b48e63796
I20260928 21:06:27.176882 22686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:27.187156 22686 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:27.188133 22686 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:27.196847 22686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b6de2669774289b08102086db80cbe: Generated new TSK 0
I20260928 21:06:27.197615 22686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:27.229525 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34859
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:27.363570 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:27.363865 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:27.363912 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:27.368772 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:27.368846 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:27.368943 22697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:27.374594 22697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:27.375787 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:27.376997 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:27.388990 22705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:27.388979 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:27.389022 22697 server_base.cc:1034] running on GCE node
W20260928 21:06:27.388979 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:27.389730 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:27.390465 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:27.391695 22697 hybrid_clock.cc:648] HybridClock initialized: now 1790629587391676 us; error 37 us; skew 500 ppm
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:27.394724 22697 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:27.395782 22697 webserver.cc:460] Webserver started at http://127.21.140.65:34611/ using document root <none> and password file <none>
I20260928 21:06:27.396407 22697 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:27.396505 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:27.396742 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:27.398967 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "12381fcd569a4ec9826399e3efe81125"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "9158dfdf3b14f692d4c3de4b598f9561"
server_key_iv: "13e67e5544ae571214c04fd325dc5d1a"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.399564 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "12381fcd569a4ec9826399e3efe81125"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "9158dfdf3b14f692d4c3de4b598f9561"
server_key_iv: "13e67e5544ae571214c04fd325dc5d1a"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.403635 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:27.406623 22712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.408093 22697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:27.408281 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "12381fcd569a4ec9826399e3efe81125"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "9158dfdf3b14f692d4c3de4b598f9561"
server_key_iv: "13e67e5544ae571214c04fd325dc5d1a"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.408432 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:27.431627 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:27.435213 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:27.435448 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:27.436088 22697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:27.436959 22697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:27.437034 22697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.437103 22697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:27.437162 22697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.446872 22697 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:40249
I20260928 21:06:27.446897 22825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:40249 every 8 connection(s)
I20260928 21:06:27.447842 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:27.457461 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22697
I20260928 21:06:27.457553 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:27.457811 22065 external_mini_cluster.cc:1428] Setting key bb72f5f5113edcb8fee9f46173a5bf4b
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:27.462412 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.449762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:35899 (local address 127.21.140.126:34859)
0928 21:06:27.449901 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.449905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.450845 (+ 940us) server_negotiation.cc:407] Connection header received
0928 21:06:27.452156 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.452160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.452209 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.452289 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:27.453551 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.454199 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.455477 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.455746 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.455841 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.458646 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:27.458654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:27.458656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:27.458681 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:27.460161 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.460709 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.460713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.460715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.460798 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.461170 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.461175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.461177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.461386 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:27.461517 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.462150 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.462260 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:06:27.463703 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.450223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34859 (local address 127.21.140.65:35899)
0928 21:06:27.450672 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:27.450717 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:27.451929 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:27.452464 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:27.452471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:27.452837 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:27.453377 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.453391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.454336 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.454339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.455262 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.455272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.455909 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.455912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.456064 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.456664 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:27.456680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:27.458475 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:27.460305 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:27.460312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:27.460328 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:27.460599 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:27.460902 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:27.460905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:27.460907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:27.461068 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:27.461538 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:27.461545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:27.461849 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 21:06:27.462102 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 21:06:27.462483 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260928 21:06:27.465076 22826 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34859
I20260928 21:06:27.465335 22826 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:27.465914 22826 heartbeater.cc:511] Master 127.21.140.126:34859 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:27.467628 22635 ts_manager.cc:194] Registered new tserver with Master: 12381fcd569a4ec9826399e3efe81125 (127.21.140.65:40249)
I20260928 21:06:27.468968 22635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:35899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:27.489645 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34859
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:27.606032 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:27.606722 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:27.606837 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:27.611155 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:27.611280 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:27.611385 22831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:27.617275 22831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:27.618626 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:27.619825 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:27.630110 22837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:27.630154 22831 server_base.cc:1034] running on GCE node
W20260928 21:06:27.630154 22839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:27.630085 22836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:27.630811 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:27.631480 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:27.632668 22831 hybrid_clock.cc:648] HybridClock initialized: now 1790629587632638 us; error 48 us; skew 500 ppm
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:27.635756 22831 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:27.636766 22831 webserver.cc:460] Webserver started at http://127.21.140.66:37669/ using document root <none> and password file <none>
I20260928 21:06:27.637308 22831 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:27.637368 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:27.637544 22831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:27.639684 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "6e8c3d7cfec145d7acf87f55df43e7c4"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "be58ed0cbd49dea06d2d061c78bf6141"
server_key_iv: "ed1dc4bf7d49b56e839c2d0ffba38b77"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.640247 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "6e8c3d7cfec145d7acf87f55df43e7c4"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "be58ed0cbd49dea06d2d061c78bf6141"
server_key_iv: "ed1dc4bf7d49b56e839c2d0ffba38b77"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.644460 22831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:27.647749 22846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.649296 22831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:27.649422 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "6e8c3d7cfec145d7acf87f55df43e7c4"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "be58ed0cbd49dea06d2d061c78bf6141"
server_key_iv: "ed1dc4bf7d49b56e839c2d0ffba38b77"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.649533 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:27.672892 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:27.676385 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:27.676625 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:27.677213 22831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:27.678097 22831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:27.678143 22831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.678185 22831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:27.678237 22831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.687695 22831 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:41707
I20260928 21:06:27.687773 22959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:41707 every 8 connection(s)
I20260928 21:06:27.688710 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:27.697340 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22831
I20260928 21:06:27.697486 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:27.697762 22065 external_mini_cluster.cc:1428] Setting key 9472c7269763f48a47072c3652954b6b
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:27.705451 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.690832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:43661 (local address 127.21.140.126:34859)
0928 21:06:27.690977 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.690981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.691895 (+ 914us) server_negotiation.cc:407] Connection header received
0928 21:06:27.693307 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.693311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.693372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.693489 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:27.694946 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.695517 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.697052 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.697322 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.697457 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.701102 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:27.701106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:27.701108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:27.701145 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:27.702955 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.703676 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.703680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.703682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.703778 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.704194 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.704199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.704201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.704422 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:27.704542 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.705122 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.705265 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:06:27.706429 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.691235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34859 (local address 127.21.140.66:43661)
0928 21:06:27.691722 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:27.691768 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:27.693057 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:27.693686 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:27.693694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:27.694203 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:27.694770 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.694785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.695685 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.695688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.696878 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.696892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.697517 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.697521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.697727 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.698562 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:27.698595 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:27.700846 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:27.703104 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:27.703112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:27.703133 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:27.703553 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:27.703887 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:27.703890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:27.703892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:27.704087 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:27.704547 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:27.704559 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:27.704853 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 21:06:27.705070 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 21:06:27.705449 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:27.708025 22960 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34859
I20260928 21:06:27.708266 22960 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:27.708799 22960 heartbeater.cc:511] Master 127.21.140.126:34859 requested a full tablet report, sending...
I20260928 21:06:27.710107 22635 ts_manager.cc:194] Registered new tserver with Master: 6e8c3d7cfec145d7acf87f55df43e7c4 (127.21.140.66:41707)
I20260928 21:06:27.710736 22635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:43661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:27.730710 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34859
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:27.849551 22965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:27.849853 22965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:27.849900 22965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:27.854216 22965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:27.854292 22965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:27.854394 22965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:27.860056 22965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:27.861265 22965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:27.862444 22965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:27.874219 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:27.874213 22971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:27.874219 22970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:27.874518 22965 server_base.cc:1034] running on GCE node
I20260928 21:06:27.874986 22965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:27.875746 22965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:27.876965 22965 hybrid_clock.cc:648] HybridClock initialized: now 1790629587876942 us; error 37 us; skew 500 ppm
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:27.879987 22965 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:27.880954 22965 webserver.cc:460] Webserver started at http://127.21.140.67:41279/ using document root <none> and password file <none>
I20260928 21:06:27.881536 22965 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:27.881598 22965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:27.881781 22965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:27.884090 22965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "63a0b9b34adf4891a34f73d2ae5a9487"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "b68e26d1deebf992c7d3df7bbbde3082"
server_key_iv: "d4b2c10c01996b2b184768406cdb6992"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.884723 22965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "63a0b9b34adf4891a34f73d2ae5a9487"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "b68e26d1deebf992c7d3df7bbbde3082"
server_key_iv: "d4b2c10c01996b2b184768406cdb6992"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.888742 22965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:06:27.891463 22980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.893036 22965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:27.893159 22965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "63a0b9b34adf4891a34f73d2ae5a9487"
format_stamp: "Formatted at 2026-09-28 21:06:27 on dist-test-slave-d5wc"
server_key: "b68e26d1deebf992c7d3df7bbbde3082"
server_key_iv: "d4b2c10c01996b2b184768406cdb6992"
server_key_version: "encryptionkey@0"
I20260928 21:06:27.893260 22965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:27.909600 22965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:27.913575 22965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:27.913748 22965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:27.914427 22965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:27.915261 22965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:27.915307 22965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.915349 22965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:27.915364 22965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:27.925329 22965 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:45203
I20260928 21:06:27.925346 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:45203 every 8 connection(s)
I20260928 21:06:27.926417 22965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:27.927970 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 22965
I20260928 21:06:27.928105 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:27.928444 22065 external_mini_cluster.cc:1428] Setting key 9ca40cfbf4c1d3b8edf9f55191f41aa8
Sep 28 21:06:27 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:27.941494 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.928534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:42925 (local address 127.21.140.126:34859)
0928 21:06:27.928684 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.928688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.929833 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 21:06:27.931457 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.931461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.931516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.931596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:27.932813 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.933364 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.934644 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.934884 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.935008 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.937624 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:27.937627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:27.937629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:27.937658 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:27.939261 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.939813 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.939817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.939820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.939905 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.940268 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.940273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.940275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.940476 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:27.940586 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.941127 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.941320 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:06:27.942521 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.929030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34859 (local address 127.21.140.67:42925)
0928 21:06:27.929635 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:27.929709 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:27.931249 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:27.931783 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:27.931791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:27.932168 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:27.932664 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.932678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.933501 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.933505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.934510 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.934519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.935045 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.935049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.935266 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.935862 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:27.935879 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:27.937424 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:27.939393 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:27.939399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:27.939416 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:27.939698 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:27.940010 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:27.940013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:27.940015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:27.940166 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:27.940595 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:27.940604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:27.940875 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 21:06:27.941089 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 21:06:27.941437 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":146,"threads_started":1}
I20260928 21:06:27.943878 23094 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34859
I20260928 21:06:27.944134 23094 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:27.944669 23094 heartbeater.cc:511] Master 127.21.140.126:34859 requested a full tablet report, sending...
I20260928 21:06:27.945760 22635 ts_manager.cc:194] Registered new tserver with Master: 63a0b9b34adf4891a34f73d2ae5a9487 (127.21.140.67:45203)
I20260928 21:06:27.946408 22635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:42925
I20260928 21:06:27.955595 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:06:27.967769 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.959653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.21.140.126:34859)
0928 21:06:27.959824 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.959828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.959844 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:06:27.959987 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.959989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.960036 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.960114 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:27.960858 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.961408 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.962353 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.962564 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.962699 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.964169 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:27.964172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:27.964174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:27.964206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:27.965908 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.966425 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.966428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.966429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.966517 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:27.966775 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:27.966777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:27.966778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:27.967104 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:27.967241 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.967482 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.967591 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 21:06:27.972870 22635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:27.975975 22635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:27.992271 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.986171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:40249 (local address 127.0.0.1:35228)
0928 21:06:27.986779 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:27.986835 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:27.986975 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:27.987767 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:27.987770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:27.987812 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:27.988237 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.988246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.989253 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.989256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.990543 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.990552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.991651 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.991655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.991774 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.991820 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 21:06:27.991938 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 21:06:27.992081 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":201,"spinlock_wait_cycles":22016,"thread_start_us":87,"threads_started":1}
I20260928 21:06:27.992591 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.985613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:41707 (local address 127.0.0.1:51832)
0928 21:06:27.987016 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:27.987030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:27.987101 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:27.987767 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:27.987770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:27.987809 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:27.988037 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.988044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.989668 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.989672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.991053 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.991063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.992181 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.992185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.992273 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.992308 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:06:27.992363 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:06:27.992444 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":58,"spinlock_wait_cycles":10880,"thread_start_us":112,"threads_started":1}
I20260928 21:06:27.992470 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.986576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.21.140.65:40249)
0928 21:06:27.987273 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.987280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.987304 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:06:27.987384 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.987389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.987602 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.987883 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:27.988368 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.989118 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.990668 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.991319 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.991423 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.991586 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.992233 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.992344 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":70,"threads_started":1}
I20260928 21:06:27.992964 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.986683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.21.140.67:45203)
0928 21:06:27.987283 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.987297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.987321 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:06:27.987396 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.987402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.987595 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.987773 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:27.988290 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.989507 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.991214 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.991861 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.991997 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.992140 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.992639 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.992812 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":168,"threads_started":1}
I20260928 21:06:27.993214 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.986206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:45203 (local address 127.0.0.1:37752)
0928 21:06:27.986904 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:27.986919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:27.987016 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:27.987762 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:27.987768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:27.987820 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:27.988112 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.988122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.989668 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.989671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.991074 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:27.991080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.992016 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.992020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:27.992107 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.992129 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:06:27.992214 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:06:27.992289 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":311,"spinlock_wait_cycles":35456,"thread_start_us":121,"threads_started":1}
I20260928 21:06:27.993340 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:27.985811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.21.140.66:41707)
0928 21:06:27.987320 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:27.987334 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:27.987351 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:06:27.987409 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:27.987413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:27.987602 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:27.988152 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:27.988327 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.989507 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.991192 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:27.992052 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:27.992887 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:27.993043 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:27.993095 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:06:27.993192 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1419,"thread_start_us":655,"threads_started":1}
I20260928 21:06:27.996035 22894 tablet_service.cc:1467] Processing CreateTablet for tablet be0b756adb6b480a96f538b6ed5ad0e5 (DEFAULT_TABLE table=test-table [id=ee4bb7825bfb4879aaf51484f2d12c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:27.996157 23028 tablet_service.cc:1467] Processing CreateTablet for tablet be0b756adb6b480a96f538b6ed5ad0e5 (DEFAULT_TABLE table=test-table [id=ee4bb7825bfb4879aaf51484f2d12c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:27.996157 22760 tablet_service.cc:1467] Processing CreateTablet for tablet be0b756adb6b480a96f538b6ed5ad0e5 (DEFAULT_TABLE table=test-table [id=ee4bb7825bfb4879aaf51484f2d12c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:27.997169 22760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0b756adb6b480a96f538b6ed5ad0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:27.997172 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0b756adb6b480a96f538b6ed5ad0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:27.997169 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0b756adb6b480a96f538b6ed5ad0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:28.003750 23111 tablet_bootstrap.cc:492] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: Bootstrap starting.
I20260928 21:06:28.005187 23113 tablet_bootstrap.cc:492] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: Bootstrap starting.
I20260928 21:06:28.006273 23111 tablet_bootstrap.cc:654] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:28.006397 23112 tablet_bootstrap.cc:492] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: Bootstrap starting.
I20260928 21:06:28.007401 23111 log.cc:824] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:28.007460 23113 tablet_bootstrap.cc:654] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:28.008268 23113 log.cc:824] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:28.008620 23112 tablet_bootstrap.cc:654] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:28.009569 23111 tablet_bootstrap.cc:492] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: No bootstrap required, opened a new log
I20260928 21:06:28.009804 23111 ts_tablet_manager.cc:1397] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:06:28.009871 23112 log.cc:824] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:28.010449 23113 tablet_bootstrap.cc:492] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: No bootstrap required, opened a new log
I20260928 21:06:28.010701 23113 ts_tablet_manager.cc:1397] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:06:28.011870 23112 tablet_bootstrap.cc:492] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: No bootstrap required, opened a new log
I20260928 21:06:28.012081 23112 ts_tablet_manager.cc:1397] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:06:28.013405 23111 raft_consensus.cc:348] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.013693 23111 raft_consensus.cc:374] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:28.013764 23111 raft_consensus.cc:729] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e8c3d7cfec145d7acf87f55df43e7c4, State: Initialized, Role: FOLLOWER
I20260928 21:06:28.014254 23111 consensus_queue.cc:260] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.014933 22960 heartbeater.cc:503] Master 127.21.140.126:34859 was elected leader, sending a full tablet report...
I20260928 21:06:28.015326 23113 raft_consensus.cc:348] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.015726 23113 raft_consensus.cc:374] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:28.015827 23113 raft_consensus.cc:729] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12381fcd569a4ec9826399e3efe81125, State: Initialized, Role: FOLLOWER
I20260928 21:06:28.016250 23112 raft_consensus.cc:348] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.016570 23112 raft_consensus.cc:374] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:28.016659 23112 raft_consensus.cc:729] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a0b9b34adf4891a34f73d2ae5a9487, State: Initialized, Role: FOLLOWER
I20260928 21:06:28.016748 23111 ts_tablet_manager.cc:1428] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260928 21:06:28.017158 23112 consensus_queue.cc:260] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.017923 23094 heartbeater.cc:503] Master 127.21.140.126:34859 was elected leader, sending a full tablet report...
I20260928 21:06:28.018402 23112 ts_tablet_manager.cc:1428] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:06:28.018314 23113 consensus_queue.cc:260] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.019074 22826 heartbeater.cc:503] Master 127.21.140.126:34859 was elected leader, sending a full tablet report...
I20260928 21:06:28.019469 23113 ts_tablet_manager.cc:1428] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260928 21:06:28.170136 23117 raft_consensus.cc:482] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:28.170352 23117 raft_consensus.cc:504] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.171536 23117 leader_election.cc:288] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63a0b9b34adf4891a34f73d2ae5a9487 (127.21.140.67:45203), 12381fcd569a4ec9826399e3efe81125 (127.21.140.65:40249)
I20260928 21:06:28.175843 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.171825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:40249 (local address 127.21.140.66:37083)
0928 21:06:28.171943 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:28.171958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:28.172061 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:28.172403 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:28.172406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:28.172426 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:28.172571 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:28.172580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.173540 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.173544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:28.174502 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:28.174512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.175359 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.175363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:28.175476 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.175502 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:06:28.175595 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:06:28.175691 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 21:06:28.175863 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.171992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:37083 (local address 127.21.140.65:40249)
0928 21:06:28.172164 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:28.172169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:28.172189 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:06:28.172239 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:28.172242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:28.172281 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:28.172390 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:28.172715 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.173398 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.174636 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.175214 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.175359 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.175420 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:28.175635 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:06:28.175735 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:06:28.176302 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.171923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:46083 (local address 127.21.140.67:45203)
0928 21:06:28.172093 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:28.172099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:28.172568 (+ 469us) server_negotiation.cc:407] Connection header received
0928 21:06:28.172628 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:28.172632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:28.172692 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:28.172765 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:28.173092 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.173730 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.174984 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.175600 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.175718 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.175764 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:28.176071 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:06:28.176169 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:06:28.176680 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.171830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:45203 (local address 127.21.140.66:46083)
0928 21:06:28.172461 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:28.172472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:28.172559 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:28.172810 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:28.172814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:28.172830 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:28.172936 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:28.172942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.173854 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.173857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:28.174801 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:28.174811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.175820 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.175824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:28.175931 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.175951 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:06:28.176493 (+ 542us) client_negotiation.cc:240] Negotiation successful
0928 21:06:28.176583 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":306,"thread_start_us":129,"threads_started":1}
I20260928 21:06:28.176820 22780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b756adb6b480a96f538b6ed5ad0e5" candidate_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12381fcd569a4ec9826399e3efe81125" is_pre_election: true
I20260928 21:06:28.177201 22780 raft_consensus.cc:2454] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e8c3d7cfec145d7acf87f55df43e7c4 in term 0.
W20260928 21:06:28.177364 23095 tablet.cc:2369] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:28.177492 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b756adb6b480a96f538b6ed5ad0e5" candidate_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" is_pre_election: true
I20260928 21:06:28.177796 22848 leader_election.cc:302] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12381fcd569a4ec9826399e3efe81125, 6e8c3d7cfec145d7acf87f55df43e7c4; no voters:
I20260928 21:06:28.177850 23048 raft_consensus.cc:2454] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e8c3d7cfec145d7acf87f55df43e7c4 in term 0.
I20260928 21:06:28.178117 23117 raft_consensus.cc:2781] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:28.178232 23117 raft_consensus.cc:482] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:28.178332 23117 raft_consensus.cc:3037] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:28.179602 23117 raft_consensus.cc:504] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.180028 23117 leader_election.cc:288] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [CANDIDATE]: Term 1 election: Requested vote from peers 63a0b9b34adf4891a34f73d2ae5a9487 (127.21.140.67:45203), 12381fcd569a4ec9826399e3efe81125 (127.21.140.65:40249)
I20260928 21:06:28.180452 23048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b756adb6b480a96f538b6ed5ad0e5" candidate_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a0b9b34adf4891a34f73d2ae5a9487"
I20260928 21:06:28.180452 22780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b756adb6b480a96f538b6ed5ad0e5" candidate_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12381fcd569a4ec9826399e3efe81125"
I20260928 21:06:28.180593 22780 raft_consensus.cc:3037] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:28.180593 23048 raft_consensus.cc:3037] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:28.182044 22780 raft_consensus.cc:2454] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e8c3d7cfec145d7acf87f55df43e7c4 in term 1.
I20260928 21:06:28.182040 23048 raft_consensus.cc:2454] T be0b756adb6b480a96f538b6ed5ad0e5 P 63a0b9b34adf4891a34f73d2ae5a9487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e8c3d7cfec145d7acf87f55df43e7c4 in term 1.
I20260928 21:06:28.182371 22847 leader_election.cc:302] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a0b9b34adf4891a34f73d2ae5a9487, 6e8c3d7cfec145d7acf87f55df43e7c4; no voters:
I20260928 21:06:28.182546 23117 raft_consensus.cc:2781] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:28.182776 23117 raft_consensus.cc:686] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [term 1 LEADER]: Becoming Leader. State: Replica: 6e8c3d7cfec145d7acf87f55df43e7c4, State: Running, Role: LEADER
I20260928 21:06:28.183135 23117 consensus_queue.cc:237] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } }
I20260928 21:06:28.186806 22635 catalog_manager.cc:5495] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e8c3d7cfec145d7acf87f55df43e7c4 (127.21.140.66). New cstate: current_term: 1 leader_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e8c3d7cfec145d7acf87f55df43e7c4" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63a0b9b34adf4891a34f73d2ae5a9487" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12381fcd569a4ec9826399e3efe81125" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40249 } health_report { overall_health: UNKNOWN } } }
W20260928 21:06:28.190150 22961 tablet.cc:2369] T be0b756adb6b480a96f538b6ed5ad0e5 P 6e8c3d7cfec145d7acf87f55df43e7c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:06:28.199736 22827 tablet.cc:2369] T be0b756adb6b480a96f538b6ed5ad0e5 P 12381fcd569a4ec9826399e3efe81125: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:28.233552 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.223434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.21.140.65:40249)
0928 21:06:28.223609 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:28.223614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:28.223749 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:06:28.223914 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:28.223917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:28.223968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:28.224063 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:28.224852 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.225361 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.226436 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.226913 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.227127 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.229879 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:28.229894 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:28.229910 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:28.229942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:28.231790 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:28.232243 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:28.232249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:28.232250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:28.232336 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:28.232580 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:28.232583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:28.232585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:28.232873 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:28.233048 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:28.233190 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 21:06:28.233300 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629588, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:28.262562 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.252921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.21.140.65:40249)
0928 21:06:28.253170 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:28.253175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:28.253427 (+ 252us) server_negotiation.cc:407] Connection header received
0928 21:06:28.253641 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:28.253645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:28.253693 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:28.253798 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:28.254673 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.255236 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.256245 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.256497 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.256628 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.259362 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:28.259365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:28.259368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:28.259399 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:28.260706 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:28.261201 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:28.261207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:28.261209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:28.261294 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:28.261587 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:28.261591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:28.261593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:28.261772 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:28.261915 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:28.262250 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 21:06:28.262383 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:28.290578 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:28.281790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.21.140.65:40249)
0928 21:06:28.282030 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:28.282036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:28.282119 (+ 83us) server_negotiation.cc:407] Connection header received
0928 21:06:28.282260 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:28.282263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:28.282317 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:28.282409 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:28.283203 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.283726 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.284646 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:28.284867 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:28.284944 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:28.287551 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:28.287554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:28.287556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:28.287580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:28.288860 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:28.289270 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:28.289272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:28.289274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:28.289345 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:28.289575 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:28.289578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:28.289579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:28.289760 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:28.289858 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:28.290272 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 21:06:28.290379 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 21:06:28.292022 22740 tablet_service.cc:2908] Rejecting scan request for tablet be0b756adb6b480a96f538b6ed5ad0e5: Uninitialized: clean time has not yet been initialized
I20260928 21:06:28.294221 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22697
I20260928 21:06:28.300539 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22831
I20260928 21:06:28.306533 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22965
I20260928 21:06:28.312549 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 22603
2026-09-28T21:06:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2544 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[23135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[23135](info): set up 2 sockets
Sep 28 21:06:28 dist-test-slave-d5wc krb5kdc[23135](info): commencing operation
krb5kdc: starting...
W20260928 21:06:29.359284 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:29 dist-test-slave-d5wc krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:29.501303 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:43551
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:43551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:29.615150 23148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:29.615497 23148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:29.615584 23148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:29.620402 23148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:29.620503 23148 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:06:29.620536 23148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:29.620594 23148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:29.620651 23148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:29.620689 23148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:29.626457 23148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:43551
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:43551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:29.627609 23148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:29.628684 23148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:29.637516 23156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:29.637517 23153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:29.637562 23154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:29.637606 23148 server_base.cc:1034] running on GCE node
I20260928 21:06:29.638518 23148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:29.639637 23148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:29.640831 23148 hybrid_clock.cc:648] HybridClock initialized: now 1790629589640814 us; error 34 us; skew 500 ppm
Sep 28 21:06:29 dist-test-slave-d5wc krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:29.643712 23148 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:29.644771 23148 webserver.cc:460] Webserver started at http://127.21.140.126:44019/ using document root <none> and password file <none>
I20260928 21:06:29.645349 23148 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:29.645447 23148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:29.645684 23148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:29.647794 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "e5f17764e4ae4784aea2de46ddadcf73"
format_stamp: "Formatted at 2026-09-28 21:06:29 on dist-test-slave-d5wc"
server_key: "c7ef1bf7fa5c2e871df1fb7ef0c94563"
server_key_iv: "3e80f3a5188f41d729c52e5ff1bd7b12"
server_key_version: "encryptionkey@0"
I20260928 21:06:29.648376 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "e5f17764e4ae4784aea2de46ddadcf73"
format_stamp: "Formatted at 2026-09-28 21:06:29 on dist-test-slave-d5wc"
server_key: "c7ef1bf7fa5c2e871df1fb7ef0c94563"
server_key_iv: "3e80f3a5188f41d729c52e5ff1bd7b12"
server_key_version: "encryptionkey@0"
I20260928 21:06:29.652303 23148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:29.654839 23163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:29.656188 23148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 21:06:29.656342 23148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "e5f17764e4ae4784aea2de46ddadcf73"
format_stamp: "Formatted at 2026-09-28 21:06:29 on dist-test-slave-d5wc"
server_key: "c7ef1bf7fa5c2e871df1fb7ef0c94563"
server_key_iv: "3e80f3a5188f41d729c52e5ff1bd7b12"
server_key_version: "encryptionkey@0"
I20260928 21:06:29.656466 23148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:29.695744 23148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:29.698875 23148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:29.699091 23148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:29.707564 23148 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:43551
I20260928 21:06:29.707580 23225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:43551 every 8 connection(s)
I20260928 21:06:29.708513 23148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:29.711213 23226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:29.716686 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: Bootstrap starting.
I20260928 21:06:29.718325 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 23148
I20260928 21:06:29.718438 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:29.718695 22065 external_mini_cluster.cc:1428] Setting key edc531ddd07604ad37dbd154dae36f49
I20260928 21:06:29.719412 23226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:29.720369 23226 log.cc:824] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:29.722718 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: No bootstrap required, opened a new log
I20260928 21:06:29.727177 23226 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f17764e4ae4784aea2de46ddadcf73" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43551 } }
I20260928 21:06:29.727448 23226 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:29.727531 23226 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f17764e4ae4784aea2de46ddadcf73, State: Initialized, Role: FOLLOWER
I20260928 21:06:29.728011 23226 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f17764e4ae4784aea2de46ddadcf73" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43551 } }
I20260928 21:06:29.728173 23226 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:29.728260 23226 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:29.728358 23226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:06:29 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:29.729552 23226 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f17764e4ae4784aea2de46ddadcf73" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43551 } }
I20260928 21:06:29.729928 23226 leader_election.cc:302] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f17764e4ae4784aea2de46ddadcf73; no voters:
I20260928 21:06:29.730357 23226 leader_election.cc:288] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:29.730489 23231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:29.731009 23231 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [term 1 LEADER]: Becoming Leader. State: Replica: e5f17764e4ae4784aea2de46ddadcf73, State: Running, Role: LEADER
I20260928 21:06:29.731355 23231 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f17764e4ae4784aea2de46ddadcf73" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43551 } }
I20260928 21:06:29.731794 23226 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:29.732604 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f17764e4ae4784aea2de46ddadcf73. Latest consensus state: current_term: 1 leader_uuid: "e5f17764e4ae4784aea2de46ddadcf73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f17764e4ae4784aea2de46ddadcf73" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43551 } } }
I20260928 21:06:29.732824 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:29.733090 23232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f17764e4ae4784aea2de46ddadcf73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f17764e4ae4784aea2de46ddadcf73" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43551 } } }
I20260928 21:06:29.733191 23232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:29.733306 23240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:29.735903 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:29.719896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.21.140.126:43551)
0928 21:06:29.720341 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:29.720356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:29.720391 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:06:29.721339 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:29.721360 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:29.721736 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:29.722375 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:29.723224 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.724790 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:29.725758 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.726526 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:29.726709 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:29.729667 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:29.729673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:29.729690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:29.729722 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:29.731809 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:29.732348 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:29.732355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:29.732361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:29.732517 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:29.732912 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:29.732915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:29.732917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:29.733467 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:29.733689 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:29.734714 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0928 21:06:29.735028 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260928 21:06:29.736301 23240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:29.742481 23240 catalog_manager.cc:1347] Generated new cluster ID: 54e9675a420c42978b18c5bd4197d11f
I20260928 21:06:29.742573 23240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:29.750949 23240 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:29.752020 23240 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:29.760969 23240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: Generated new TSK 0
I20260928 21:06:29.761857 23240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:29.797567 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:43551
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:29.910359 23252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:29.910631 23252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:29.910704 23252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:29.914938 23252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:29.915005 23252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:29.915077 23252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:06:29.915144 23252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:29.920454 23252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:29.921578 23252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:29.922683 23252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:29.932650 23258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:29.932662 23260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:29.932662 23257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:29.932768 23252 server_base.cc:1034] running on GCE node
I20260928 21:06:29.933445 23252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:29.934309 23252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:29.935537 23252 hybrid_clock.cc:648] HybridClock initialized: now 1790629589935521 us; error 28 us; skew 500 ppm
Sep 28 21:06:29 dist-test-slave-d5wc krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:29.938652 23252 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:29.939677 23252 webserver.cc:460] Webserver started at http://127.21.140.65:37175/ using document root <none> and password file <none>
I20260928 21:06:29.940285 23252 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:29.940385 23252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:29.940634 23252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:29.942916 23252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "5d70adad7130475eab279c5dfb6c0db3"
format_stamp: "Formatted at 2026-09-28 21:06:29 on dist-test-slave-d5wc"
server_key: "39880fe72def720590e5b58ae7c19c0b"
server_key_iv: "47413d4c656e36aeca57391f11c09970"
server_key_version: "encryptionkey@0"
I20260928 21:06:29.943559 23252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "5d70adad7130475eab279c5dfb6c0db3"
format_stamp: "Formatted at 2026-09-28 21:06:29 on dist-test-slave-d5wc"
server_key: "39880fe72def720590e5b58ae7c19c0b"
server_key_iv: "47413d4c656e36aeca57391f11c09970"
server_key_version: "encryptionkey@0"
I20260928 21:06:29.947587 23252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:29.950119 23267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:29.951536 23252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:29.951673 23252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "5d70adad7130475eab279c5dfb6c0db3"
format_stamp: "Formatted at 2026-09-28 21:06:29 on dist-test-slave-d5wc"
server_key: "39880fe72def720590e5b58ae7c19c0b"
server_key_iv: "47413d4c656e36aeca57391f11c09970"
server_key_version: "encryptionkey@0"
I20260928 21:06:29.951792 23252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:29.970494 23252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:29.974169 23252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:29.974390 23252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:29.975853 23252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:29.975931 23252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:29.975999 23252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:29.976051 23252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:06:29 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:29.987671 23252 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:45929
I20260928 21:06:29.987941 23382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:45929 every 8 connection(s)
I20260928 21:06:29.988806 23252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:29.991231 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:29.977079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:51009 (local address 127.21.140.126:43551)
0928 21:06:29.977235 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:29.977239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:29.978475 (+ 1236us) server_negotiation.cc:407] Connection header received
0928 21:06:29.979903 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:29.979910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:29.979982 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:29.980088 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:29.981407 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.982162 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:29.983597 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.983875 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:29.984008 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:29.987232 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:29.987238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:29.987242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:29.987276 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:29.989074 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:29.989674 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:29.989678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:29.989680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:29.989772 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:29.990193 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:29.990197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:29.990198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:29.990402 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:29.990503 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:29.990966 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 21:06:29.991099 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:06:29.992061 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:29.977647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.21.140.65:51009)
0928 21:06:29.978289 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:29.978336 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:29.979603 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:29.980256 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:29.980264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:29.980698 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:29.981237 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:29.981251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.982317 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:29.982323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:29.983459 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:29.983470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.984080 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:29.984085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:29.984334 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:29.985077 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:29.985095 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:29.987039 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:29.989198 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:29.989208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:29.989227 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:29.989573 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:29.989879 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:29.989883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:29.989887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:29.990087 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:29.990499 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:29.990504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:29.990729 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 21:06:29.990876 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 21:06:29.991171 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":145,"threads_started":1}
I20260928 21:06:29.993384 23383 heartbeater.cc:347] Connected to a master server at 127.21.140.126:43551
I20260928 21:06:29.993551 23383 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:29.994195 23383 heartbeater.cc:511] Master 127.21.140.126:43551 requested a full tablet report, sending...
I20260928 21:06:29.994372 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 23252
I20260928 21:06:29.994490 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:29.994774 22065 external_mini_cluster.cc:1428] Setting key 13a225cd07c5582fbacf9fa0cdebb621
I20260928 21:06:29.996065 23179 ts_manager.cc:194] Registered new tserver with Master: 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929)
I20260928 21:06:29.997236 23179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:51009
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:30.006757 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:29.996185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.21.140.65:45679)
0928 21:06:29.998127 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:29.998146 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:29.998273 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:29.998562 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:29.998567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:29.998870 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:29.999057 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:29.999067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.000097 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.000104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.001249 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.001262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.001734 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.001742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.001886 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.002477 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:30.002511 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:30.003049 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:30.004840 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.004844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.004846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.005159 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.005538 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.005541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.005544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.005663 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.006288 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:30.006290 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:30.006380 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.006476 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.006596 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1827,"thread_start_us":97,"threads_started":1}
I20260928 21:06:30.006922 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:29.997797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:45679 (local address 127.21.140.126:43551)
0928 21:06:29.998037 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:29.998043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:29.998232 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:06:29.998383 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:29.998386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:29.998433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:29.998522 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:29.999206 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:29.999937 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.001380 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.001599 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.001690 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.003199 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:30.003202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:30.003204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:30.003226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:30.004718 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.005331 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.005335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.005336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.005428 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.005813 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.005817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.005819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.006159 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:30.006249 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.006580 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.006701 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:30.025667 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:43551
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:30.141443 23392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:30.141719 23392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:30.141767 23392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:30.146155 23392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:30.146231 23392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:30.146275 23392 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:06:30.146309 23392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:30.152297 23392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:30.153396 23392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:30.154915 23392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:30.165132 23400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:30.165155 23397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:30.165196 23392 server_base.cc:1034] running on GCE node
W20260928 21:06:30.165155 23398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:30.165881 23392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:30.166690 23392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:30.167891 23392 hybrid_clock.cc:648] HybridClock initialized: now 1790629590167869 us; error 33 us; skew 500 ppm
Sep 28 21:06:30 dist-test-slave-d5wc krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:30.177775 23392 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:30.178884 23392 webserver.cc:460] Webserver started at http://127.21.140.66:37671/ using document root <none> and password file <none>
I20260928 21:06:30.179646 23392 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:30.179774 23392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:30.180083 23392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:30.182442 23392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "1447cd7dbb6b4ab08ec90df4f04a1295"
format_stamp: "Formatted at 2026-09-28 21:06:30 on dist-test-slave-d5wc"
server_key: "90ed4ba160f139ddf1e6d133a6344a63"
server_key_iv: "43a7187b8aec7f6334a9c04cdfb41144"
server_key_version: "encryptionkey@0"
I20260928 21:06:30.183028 23392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "1447cd7dbb6b4ab08ec90df4f04a1295"
format_stamp: "Formatted at 2026-09-28 21:06:30 on dist-test-slave-d5wc"
server_key: "90ed4ba160f139ddf1e6d133a6344a63"
server_key_iv: "43a7187b8aec7f6334a9c04cdfb41144"
server_key_version: "encryptionkey@0"
I20260928 21:06:30.186875 23392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:30.189474 23407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:30.190798 23392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:06:30.190943 23392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "1447cd7dbb6b4ab08ec90df4f04a1295"
format_stamp: "Formatted at 2026-09-28 21:06:30 on dist-test-slave-d5wc"
server_key: "90ed4ba160f139ddf1e6d133a6344a63"
server_key_iv: "43a7187b8aec7f6334a9c04cdfb41144"
server_key_version: "encryptionkey@0"
I20260928 21:06:30.191059 23392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:30.215145 23392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:30.218274 23392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:30.218490 23392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:30.219986 23392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:30.220064 23392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:30.220135 23392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:30.220180 23392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:30.231046 23392 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:44637
I20260928 21:06:30.231065 23522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:44637 every 8 connection(s)
I20260928 21:06:30.231977 23392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:30.232735 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 23392
I20260928 21:06:30.232847 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:30.233194 22065 external_mini_cluster.cc:1428] Setting key bac7618b4adb13f7dbccfb198c1e6049
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Sep 28 21:06:30 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:30.251863 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.220920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:39097 (local address 127.21.140.126:43551)
0928 21:06:30.221070 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.221074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.222265 (+ 1191us) server_negotiation.cc:407] Connection header received
0928 21:06:30.223468 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.223472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.223534 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.223642 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:30.225243 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.226145 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.227508 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.227751 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.227866 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.247535 (+ 19669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:30.247539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:30.247541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:30.247573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:30.249275 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.249929 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.249934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.249938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.250099 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.250512 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.250517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.250519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.250747 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:30.250872 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.251526 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.251673 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:06:30.253014 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.221554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.21.140.66:39097)
0928 21:06:30.222081 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.222126 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.223138 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.223817 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.223826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.224336 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:30.225059 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.225083 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.226274 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.226277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.227379 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.227388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.227918 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.227922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.228123 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.245317 (+ 17194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:30.245341 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:30.247255 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:30.249442 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.249449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.249469 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.249798 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.250214 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.250218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.250220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.250396 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.250870 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:30.250879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:30.251189 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.251476 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.251893 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
I20260928 21:06:30.255326 23523 heartbeater.cc:347] Connected to a master server at 127.21.140.126:43551
I20260928 21:06:30.255577 23523 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:30.256699 23523 heartbeater.cc:511] Master 127.21.140.126:43551 requested a full tablet report, sending...
I20260928 21:06:30.258356 23180 ts_manager.cc:194] Registered new tserver with Master: 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637)
I20260928 21:06:30.259347 23180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:39097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:30.263798 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:43551
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:06:30.269198 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.258248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:50681 (local address 127.21.140.126:43551)
0928 21:06:30.258416 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.258421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.259014 (+ 593us) server_negotiation.cc:407] Connection header received
0928 21:06:30.259093 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.259097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.259154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.259241 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:30.259902 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.260810 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.262688 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.263161 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.263244 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.264651 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:30.264655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:30.264658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:30.264685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:30.267192 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.267686 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.267690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.267692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.267787 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.268106 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.268111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.268113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.268355 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:30.268478 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.268785 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.268906 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 21:06:30.269210 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.258097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.21.140.66:50681)
0928 21:06:30.258473 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.258504 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.258624 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.259305 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.259308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.259564 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:30.259767 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.259774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.260949 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.260953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.262524 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.262538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.263393 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.263396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.263470 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.264121 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:30.264134 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:30.264493 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:30.267335 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.267339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.267341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.267570 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.267888 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.267893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.267895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.267994 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.268480 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:30.268484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:30.268597 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.268722 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.268894 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":235,"thread_start_us":104,"threads_started":1}
W20260928 21:06:30.396158 23532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:30.396494 23532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:30.396571 23532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:30.400769 23532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:30.400884 23532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:30.400962 23532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:06:30.401064 23532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:30.406746 23532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:30.407976 23532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:30.409101 23532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:30.419415 23538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:30.419409 23540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:30.419409 23537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:30.419664 23532 server_base.cc:1034] running on GCE node
I20260928 21:06:30.420321 23532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:30.421061 23532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:30.422276 23532 hybrid_clock.cc:648] HybridClock initialized: now 1790629590422250 us; error 29 us; skew 500 ppm
Sep 28 21:06:30 dist-test-slave-d5wc krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:30.425527 23532 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:30.426617 23532 webserver.cc:460] Webserver started at http://127.21.140.67:44127/ using document root <none> and password file <none>
I20260928 21:06:30.427196 23532 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:30.427260 23532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:30.427440 23532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:30.429688 23532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "b18adfb839fa4409bec450dd9ced5a2f"
format_stamp: "Formatted at 2026-09-28 21:06:30 on dist-test-slave-d5wc"
server_key: "f34c25e9579bd8f4d322770feb0ea1c5"
server_key_iv: "5f98766f30343a3454e9216728c809e7"
server_key_version: "encryptionkey@0"
I20260928 21:06:30.430385 23532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "b18adfb839fa4409bec450dd9ced5a2f"
format_stamp: "Formatted at 2026-09-28 21:06:30 on dist-test-slave-d5wc"
server_key: "f34c25e9579bd8f4d322770feb0ea1c5"
server_key_iv: "5f98766f30343a3454e9216728c809e7"
server_key_version: "encryptionkey@0"
I20260928 21:06:30.434909 23532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:30.437641 23547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:30.439029 23532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:06:30.439131 23532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "b18adfb839fa4409bec450dd9ced5a2f"
format_stamp: "Formatted at 2026-09-28 21:06:30 on dist-test-slave-d5wc"
server_key: "f34c25e9579bd8f4d322770feb0ea1c5"
server_key_iv: "5f98766f30343a3454e9216728c809e7"
server_key_version: "encryptionkey@0"
I20260928 21:06:30.439225 23532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:30.458662 23532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:30.462134 23532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:30.462302 23532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:30.463788 23532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:30.463865 23532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:30.463935 23532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:30.463989 23532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:06:30 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:30.475981 23532 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:38165
I20260928 21:06:30.476047 23662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:38165 every 8 connection(s)
I20260928 21:06:30.476994 23532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:30.480166 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.464808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:46661 (local address 127.21.140.126:43551)
0928 21:06:30.464953 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.464957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.466114 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 21:06:30.467497 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.467502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.467574 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.467679 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:30.469585 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.470504 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.472136 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.472448 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.472602 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.476280 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:30.476284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:30.476286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:30.476313 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:30.478140 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.478729 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.478732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.478733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.478814 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.479220 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.479222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.479223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.479405 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:30.479480 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.479943 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.480046 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:06:30.481106 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.465339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.21.140.67:46661)
0928 21:06:30.465859 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.465904 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.467184 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.467928 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.467937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.468501 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:30.469391 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.469412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.470670 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.470673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.471975 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.471987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.472646 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.472650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.472855 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.473543 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:30.473571 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:30.476081 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:30.478265 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.478270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.478306 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.478628 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.478929 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.478932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.478933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.479122 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.479499 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:30.479504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:30.479736 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.479895 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.480202 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":158,"threads_started":1}
I20260928 21:06:30.482318 23663 heartbeater.cc:347] Connected to a master server at 127.21.140.126:43551
I20260928 21:06:30.482482 23663 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:30.483073 23663 heartbeater.cc:511] Master 127.21.140.126:43551 requested a full tablet report, sending...
I20260928 21:06:30.483675 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 23532
I20260928 21:06:30.483796 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:30.484079 22065 external_mini_cluster.cc:1428] Setting key d9660fc37db1f2def9085d25c1248bef
I20260928 21:06:30.484109 23179 ts_manager.cc:194] Registered new tserver with Master: b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67:38165)
I20260928 21:06:30.484926 23179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:46661
I20260928 21:06:30.486100 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:06:30.493449 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.484754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:46185 (local address 127.21.140.126:43551)
0928 21:06:30.484974 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.484979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.485200 (+ 221us) server_negotiation.cc:407] Connection header received
0928 21:06:30.485381 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.485384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.485425 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.485529 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:30.486633 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.487425 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.488468 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.488684 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.488768 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.490176 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:30.490179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:30.490182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:30.490204 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:30.491643 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.492147 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.492150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.492152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.492224 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.492556 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.492559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.492561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.492781 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:30.492893 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.493190 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.493297 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 21:06:30.493464 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.484682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.21.140.67:46185)
0928 21:06:30.485127 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.485142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.485268 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.486080 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.486083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.486320 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:30.486499 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.486506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.487561 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.487564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.488337 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.488350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.488800 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.488805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.488924 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.489509 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:30.489528 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:30.490040 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:30.491755 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.491760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.491762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.492039 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.492318 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:30.492324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:30.492329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:30.492434 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:30.492884 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:30.492887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:30.492978 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.493135 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.493287 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
Sep 28 21:06:30 dist-test-slave-d5wc krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:06:30 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:30.513031 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.505019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.21.140.126:43551)
0928 21:06:30.505203 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.505210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.505228 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:06:30.505282 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.505287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.505348 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.505447 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:30.506273 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.506791 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.507631 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.507845 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.507927 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.510370 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:30.510374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:30.510377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:30.510405 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:30.511689 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.512054 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.512056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.512058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.512126 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:30.512377 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:30.512379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:30.512381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:30.512538 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:30.512637 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.512825 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.512911 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:06:30.515913 23179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:30.518939 23179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:30.533538 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.527378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:45929 (local address 127.0.0.1:59614)
0928 21:06:30.527876 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.527924 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.528066 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.529132 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.529138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.529170 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.529309 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.529316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.530529 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.530534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.531778 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.531789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.532928 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.532932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.533077 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.533127 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.533241 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.533371 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":3,"spinlock_wait_cycles":5248,"thread_start_us":92,"threads_started":1}
I20260928 21:06:30.533718 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.527518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:44637 (local address 127.0.0.1:47170)
0928 21:06:30.527876 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.527925 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.528063 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.529378 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.529381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.529398 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.529507 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.529518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.531081 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.531085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.532268 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.532277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.533382 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.533386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.533478 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.533496 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.533538 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.533594 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":23,"spinlock_wait_cycles":3712,"thread_start_us":99,"threads_started":1}
I20260928 21:06:30.533903 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.527494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.21.140.65:45929)
0928 21:06:30.528647 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.528655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.528681 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:06:30.528758 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.528764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.528988 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.529138 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.529454 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.530416 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.531945 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.532792 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.532894 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.533095 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.533465 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.533572 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":75,"threads_started":1}
I20260928 21:06:30.534230 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.528636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.21.140.67:38165)
0928 21:06:30.529056 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.529069 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.529092 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:06:30.529160 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.529165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.529344 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.529483 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.529836 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.530967 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.532355 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.533099 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.533219 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.533378 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.533617 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.534076 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":118,"threads_started":1}
I20260928 21:06:30.534436 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.528230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.21.140.66:44637)
0928 21:06:30.528921 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.528931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.528952 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:06:30.529017 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.529023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.529192 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.529395 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.529747 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.530934 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.532401 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.533256 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.533908 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.534137 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.534199 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.534295 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":106,"threads_started":1}
I20260928 21:06:30.534473 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.528205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38165 (local address 127.0.0.1:48176)
0928 21:06:30.528431 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.528447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.528532 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.529606 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.529609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.529625 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.529714 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.529719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.531096 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.531099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.532228 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.532237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.533231 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.533234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.533349 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.533379 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.533443 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.533511 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":61,"thread_start_us":105,"threads_started":1}
I20260928 21:06:30.536948 23597 tablet_service.cc:1467] Processing CreateTablet for tablet 2453e27f765a4ee6b6fb57983ac1cc62 (DEFAULT_TABLE table=test-table [id=698d4901bd944bd59464189f015fb8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:30.536980 23317 tablet_service.cc:1467] Processing CreateTablet for tablet 2453e27f765a4ee6b6fb57983ac1cc62 (DEFAULT_TABLE table=test-table [id=698d4901bd944bd59464189f015fb8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:30.536983 23457 tablet_service.cc:1467] Processing CreateTablet for tablet 2453e27f765a4ee6b6fb57983ac1cc62 (DEFAULT_TABLE table=test-table [id=698d4901bd944bd59464189f015fb8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:30.537997 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2453e27f765a4ee6b6fb57983ac1cc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:30.537992 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2453e27f765a4ee6b6fb57983ac1cc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:30.537951 23597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2453e27f765a4ee6b6fb57983ac1cc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:30.545874 23687 tablet_bootstrap.cc:492] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Bootstrap starting.
I20260928 21:06:30.545918 23686 tablet_bootstrap.cc:492] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Bootstrap starting.
I20260928 21:06:30.546277 23688 tablet_bootstrap.cc:492] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Bootstrap starting.
I20260928 21:06:30.547984 23688 tablet_bootstrap.cc:654] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:30.548084 23687 tablet_bootstrap.cc:654] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:30.548084 23686 tablet_bootstrap.cc:654] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:30.548915 23687 log.cc:824] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:30.548915 23688 log.cc:824] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:30.548915 23686 log.cc:824] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:30.550954 23687 tablet_bootstrap.cc:492] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: No bootstrap required, opened a new log
I20260928 21:06:30.551162 23687 ts_tablet_manager.cc:1397] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:06:30.551231 23688 tablet_bootstrap.cc:492] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: No bootstrap required, opened a new log
I20260928 21:06:30.551397 23688 ts_tablet_manager.cc:1397] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:06:30.551589 23686 tablet_bootstrap.cc:492] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: No bootstrap required, opened a new log
I20260928 21:06:30.551795 23686 ts_tablet_manager.cc:1397] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:06:30.555262 23686 raft_consensus.cc:348] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.555608 23686 raft_consensus.cc:374] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:30.555461 23688 raft_consensus.cc:348] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.555670 23686 raft_consensus.cc:729] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18adfb839fa4409bec450dd9ced5a2f, State: Initialized, Role: FOLLOWER
I20260928 21:06:30.555718 23688 raft_consensus.cc:374] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:30.555603 23687 raft_consensus.cc:348] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.555765 23688 raft_consensus.cc:729] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447cd7dbb6b4ab08ec90df4f04a1295, State: Initialized, Role: FOLLOWER
I20260928 21:06:30.555828 23687 raft_consensus.cc:374] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:30.555878 23687 raft_consensus.cc:729] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d70adad7130475eab279c5dfb6c0db3, State: Initialized, Role: FOLLOWER
I20260928 21:06:30.556205 23688 consensus_queue.cc:260] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.556238 23686 consensus_queue.cc:260] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.556411 23687 consensus_queue.cc:260] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.557010 23523 heartbeater.cc:503] Master 127.21.140.126:43551 was elected leader, sending a full tablet report...
I20260928 21:06:30.557232 23688 ts_tablet_manager.cc:1428] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:06:30.557304 23686 ts_tablet_manager.cc:1428] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260928 21:06:30.557655 23663 heartbeater.cc:503] Master 127.21.140.126:43551 was elected leader, sending a full tablet report...
I20260928 21:06:30.557778 23687 ts_tablet_manager.cc:1428] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:06:30.558092 23383 heartbeater.cc:503] Master 127.21.140.126:43551 was elected leader, sending a full tablet report...
I20260928 21:06:30.727723 23693 raft_consensus.cc:482] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:30.727937 23693 raft_consensus.cc:504] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
W20260928 21:06:30.728657 23664 tablet.cc:2369] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:30.729303 23693 leader_election.cc:288] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929), 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637)
I20260928 21:06:30.733268 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.729562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:44637 (local address 127.21.140.67:33857)
0928 21:06:30.729700 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.729719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.729821 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.730175 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.730178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.730196 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.730313 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.730318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.731068 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.731071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.731941 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.731951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.732810 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.732814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.732936 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.732954 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.733045 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.733123 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 21:06:30.733228 23524 tablet.cc:2369] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:30.733492 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.729758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:33857 (local address 127.21.140.66:44637)
0928 21:06:30.729885 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.729890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.729908 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:06:30.729985 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.729988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.730040 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.730153 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.730422 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.730943 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.732138 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.732689 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.732818 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.732877 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.733060 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.733131 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:06:30.734097 23477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2453e27f765a4ee6b6fb57983ac1cc62" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" is_pre_election: true
I20260928 21:06:30.734246 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.729655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:45929 (local address 127.21.140.67:48169)
0928 21:06:30.730044 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.730058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.730195 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.730437 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.730440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.730454 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.730921 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.730935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.731812 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.731819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.732839 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.732849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.733788 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.733794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.733892 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.733915 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.734040 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.734113 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":149,"threads_started":1}
I20260928 21:06:30.734285 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.729797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:48169 (local address 127.21.140.65:45929)
0928 21:06:30.730045 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.730051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.730143 (+ 92us) server_negotiation.cc:407] Connection header received
0928 21:06:30.730302 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.730305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.730344 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.730428 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.731105 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.731701 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.732961 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.733634 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.733769 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.733829 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.734057 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.734132 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 21:06:30.734429 23477 raft_consensus.cc:2454] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 0.
I20260928 21:06:30.734949 23550 leader_election.cc:302] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447cd7dbb6b4ab08ec90df4f04a1295, b18adfb839fa4409bec450dd9ced5a2f; no voters:
I20260928 21:06:30.734999 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2453e27f765a4ee6b6fb57983ac1cc62" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d70adad7130475eab279c5dfb6c0db3" is_pre_election: true
I20260928 21:06:30.735261 23693 raft_consensus.cc:2781] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:30.735291 23337 raft_consensus.cc:2454] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 0.
I20260928 21:06:30.735363 23693 raft_consensus.cc:482] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:30.735426 23693 raft_consensus.cc:3037] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:30.736555 23693 raft_consensus.cc:504] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.736948 23693 leader_election.cc:288] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 election: Requested vote from peers 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929), 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637)
I20260928 21:06:30.737357 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2453e27f765a4ee6b6fb57983ac1cc62" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d70adad7130475eab279c5dfb6c0db3"
I20260928 21:06:30.737357 23477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2453e27f765a4ee6b6fb57983ac1cc62" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295"
I20260928 21:06:30.737491 23477 raft_consensus.cc:3037] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:30.737501 23337 raft_consensus.cc:3037] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:30.738847 23337 raft_consensus.cc:2454] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 1.
I20260928 21:06:30.738847 23477 raft_consensus.cc:2454] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 1.
I20260928 21:06:30.739166 23549 leader_election.cc:302] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1447cd7dbb6b4ab08ec90df4f04a1295, 5d70adad7130475eab279c5dfb6c0db3, b18adfb839fa4409bec450dd9ced5a2f; no voters:
I20260928 21:06:30.739378 23693 raft_consensus.cc:2781] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:30.739614 23693 raft_consensus.cc:686] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 1 LEADER]: Becoming Leader. State: Replica: b18adfb839fa4409bec450dd9ced5a2f, State: Running, Role: LEADER
I20260928 21:06:30.739975 23693 consensus_queue.cc:237] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
W20260928 21:06:30.740314 23384 tablet.cc:2369] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:30.743642 23179 catalog_manager.cc:5495] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f reported cstate change: term changed from 0 to 1, leader changed from <none> to b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67). New cstate: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } health_report { overall_health: HEALTHY } } }
I20260928 21:06:30.779413 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.775685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.21.140.67:38165)
0928 21:06:30.775800 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.775804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.775818 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:06:30.776013 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.776056 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.776111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.776192 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:30.776497 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.776997 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.777923 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.778236 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.778321 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.778620 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:30.779057 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:30.779146 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.779252 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.779305 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:06:30.785486 23337 raft_consensus.cc:1261] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Refusing update from remote peer b18adfb839fa4409bec450dd9ced5a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:30.785486 23477 raft_consensus.cc:1261] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Refusing update from remote peer b18adfb839fa4409bec450dd9ced5a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:30.786291 23696 consensus_queue.cc:1035] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:30.786523 23693 consensus_queue.cc:1035] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:30.794606 23701 mvcc.cc:205] Tried to move back new op lower bound from 7334418803849596928 to 7334418803674021888. Current Snapshot: MvccSnapshot[applied={T|T < 7334418803849596928}]
I20260928 21:06:30.807742 23179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:30.807951 23179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:30.810565 23179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: Sending DeleteTablet for 3 replicas of tablet 2453e27f765a4ee6b6fb57983ac1cc62
I20260928 21:06:30.811472 23317 tablet_service.cc:1514] Processing DeleteTablet for tablet 2453e27f765a4ee6b6fb57983ac1cc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:30 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:59614
I20260928 21:06:30.811482 23457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2453e27f765a4ee6b6fb57983ac1cc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:30 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:47170
I20260928 21:06:30.812177 23597 tablet_service.cc:1514] Processing DeleteTablet for tablet 2453e27f765a4ee6b6fb57983ac1cc62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:30 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 21:06:30.812245 23709 tablet_replica.cc:330] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: stopping tablet replica
I20260928 21:06:30.812219 23179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:06:30.812620 23709 raft_consensus.cc:2229] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:30.812654 23708 tablet_replica.cc:330] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: stopping tablet replica
W20260928 21:06:30.812672 23179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:30.812940 23709 raft_consensus.cc:2258] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:30.812966 23708 raft_consensus.cc:2229] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:30.813269 23708 raft_consensus.cc:2258] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:30.814575 23709 ts_tablet_manager.cc:1905] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:30.815234 23708 ts_tablet_manager.cc:1905] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:30.817380 23709 ts_tablet_manager.cc:1918] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:30.817477 23709 log.cc:1196] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/2453e27f765a4ee6b6fb57983ac1cc62
I20260928 21:06:30.817803 23709 ts_tablet_manager.cc:1939] T 2453e27f765a4ee6b6fb57983ac1cc62 P 5d70adad7130475eab279c5dfb6c0db3: Deleting consensus metadata
I20260928 21:06:30.818146 23708 ts_tablet_manager.cc:1918] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:30.818234 23708 log.cc:1196] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/2453e27f765a4ee6b6fb57983ac1cc62
I20260928 21:06:30.818375 23457 tablet_service.cc:1467] Processing CreateTablet for tablet 643d6931e687410eb03af075be811692 (DEFAULT_TABLE table=test-table [id=c551d31c678841079c0d59943e437eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:30.818645 23708 ts_tablet_manager.cc:1939] T 2453e27f765a4ee6b6fb57983ac1cc62 P 1447cd7dbb6b4ab08ec90df4f04a1295: Deleting consensus metadata
I20260928 21:06:30.818696 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 643d6931e687410eb03af075be811692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:30.819105 23710 tablet_replica.cc:330] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: stopping tablet replica
I20260928 21:06:30.819402 23710 raft_consensus.cc:2229] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:30.819785 23710 raft_consensus.cc:2258] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:30.819971 23166 catalog_manager.cc:4841] TS 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637): tablet 2453e27f765a4ee6b6fb57983ac1cc62 (table test-table [id=698d4901bd944bd59464189f015fb8fa]) successfully deleted
I20260928 21:06:30.820500 23597 tablet_service.cc:1467] Processing CreateTablet for tablet 643d6931e687410eb03af075be811692 (DEFAULT_TABLE table=test-table [id=c551d31c678841079c0d59943e437eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:30.820746 23597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 643d6931e687410eb03af075be811692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:30.820868 23688 tablet_bootstrap.cc:492] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Bootstrap starting.
I20260928 21:06:30.821254 23710 ts_tablet_manager.cc:1905] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:30.821803 23688 tablet_bootstrap.cc:654] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:30.823276 23686 tablet_bootstrap.cc:492] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Bootstrap starting.
I20260928 21:06:30.823626 23710 ts_tablet_manager.cc:1918] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:30.823642 23165 catalog_manager.cc:4841] TS 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929): tablet 2453e27f765a4ee6b6fb57983ac1cc62 (table test-table [id=698d4901bd944bd59464189f015fb8fa]) successfully deleted
I20260928 21:06:30.823711 23710 log.cc:1196] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/2453e27f765a4ee6b6fb57983ac1cc62
I20260928 21:06:30.824002 23710 ts_tablet_manager.cc:1939] T 2453e27f765a4ee6b6fb57983ac1cc62 P b18adfb839fa4409bec450dd9ced5a2f: Deleting consensus metadata
I20260928 21:06:30.824234 23686 tablet_bootstrap.cc:654] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:30.824623 23317 tablet_service.cc:1467] Processing CreateTablet for tablet 643d6931e687410eb03af075be811692 (DEFAULT_TABLE table=test-table [id=c551d31c678841079c0d59943e437eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:30.824894 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 643d6931e687410eb03af075be811692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:30.825410 23686 tablet_bootstrap.cc:492] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: No bootstrap required, opened a new log
I20260928 21:06:30.825500 23686 ts_tablet_manager.cc:1397] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 21:06:30.825940 23686 raft_consensus.cc:348] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.826088 23686 raft_consensus.cc:374] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:30.826125 23686 raft_consensus.cc:729] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18adfb839fa4409bec450dd9ced5a2f, State: Initialized, Role: FOLLOWER
I20260928 21:06:30.826257 23686 consensus_queue.cc:260] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.826301 23166 catalog_manager.cc:4841] TS b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67:38165): tablet 2453e27f765a4ee6b6fb57983ac1cc62 (table test-table [id=698d4901bd944bd59464189f015fb8fa]) successfully deleted
I20260928 21:06:30.826512 23686 ts_tablet_manager.cc:1428] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:06:30.827322 23687 tablet_bootstrap.cc:492] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Bootstrap starting.
I20260928 21:06:30.828101 23688 tablet_bootstrap.cc:492] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: No bootstrap required, opened a new log
I20260928 21:06:30.828202 23688 ts_tablet_manager.cc:1397] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260928 21:06:30.828348 23687 tablet_bootstrap.cc:654] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:30.828588 23688 raft_consensus.cc:348] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.828675 23688 raft_consensus.cc:374] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:30.828696 23688 raft_consensus.cc:729] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447cd7dbb6b4ab08ec90df4f04a1295, State: Initialized, Role: FOLLOWER
I20260928 21:06:30.828825 23688 consensus_queue.cc:260] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.829093 23688 ts_tablet_manager.cc:1428] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:06:30.829793 23687 tablet_bootstrap.cc:492] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: No bootstrap required, opened a new log
I20260928 21:06:30.829898 23687 ts_tablet_manager.cc:1397] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:30.830448 23687 raft_consensus.cc:348] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.830562 23687 raft_consensus.cc:374] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:30.830597 23687 raft_consensus.cc:729] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d70adad7130475eab279c5dfb6c0db3, State: Initialized, Role: FOLLOWER
I20260928 21:06:30.830714 23687 consensus_queue.cc:260] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.830983 23687 ts_tablet_manager.cc:1428] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:30.942421 23692 raft_consensus.cc:482] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:30.942605 23692 raft_consensus.cc:504] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.943840 23692 leader_election.cc:288] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929), b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67:38165)
I20260928 21:06:30.948675 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.944272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:35533 (local address 127.21.140.67:38165)
0928 21:06:30.944438 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.944444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.944615 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:06:30.944862 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.944865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.944927 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.945032 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.945300 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.946163 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.947652 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.948040 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.948188 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.948249 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.948451 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.948533 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:06:30.948804 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.944568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:47809 (local address 127.21.140.65:45929)
0928 21:06:30.944707 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:30.944712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:30.944727 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:06:30.944925 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:30.944929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:30.944988 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:30.945062 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:30.945427 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.945924 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.947625 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.948265 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.948382 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.948430 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:30.948604 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 21:06:30.948678 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:06:30.949105 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.944161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38165 (local address 127.21.140.66:35533)
0928 21:06:30.944522 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.944536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.944698 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.945041 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.945044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.945059 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.945187 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.945194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.946292 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.946295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.947489 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.947495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.948190 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.948193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.948298 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.948322 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.948888 (+ 566us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.948968 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":26,"thread_start_us":88,"threads_started":1}
I20260928 21:06:30.949105 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:30.944167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:45929 (local address 127.21.140.66:47809)
0928 21:06:30.944633 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:30.944648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:30.944754 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:30.945157 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:30.945160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:30.945183 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:30.945300 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.945307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.946096 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.946101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.947477 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:30.947489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:30.948381 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:30.948386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:30.948487 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:30.948506 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:06:30.948574 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:06:30.948639 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":150,"thread_start_us":98,"threads_started":1}
I20260928 21:06:30.949627 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "643d6931e687410eb03af075be811692" candidate_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d70adad7130475eab279c5dfb6c0db3" is_pre_election: true
I20260928 21:06:30.949811 23337 raft_consensus.cc:2454] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1447cd7dbb6b4ab08ec90df4f04a1295 in term 0.
I20260928 21:06:30.949761 23617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "643d6931e687410eb03af075be811692" candidate_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18adfb839fa4409bec450dd9ced5a2f" is_pre_election: true
I20260928 21:06:30.950007 23617 raft_consensus.cc:2454] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1447cd7dbb6b4ab08ec90df4f04a1295 in term 0.
I20260928 21:06:30.950189 23409 leader_election.cc:302] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447cd7dbb6b4ab08ec90df4f04a1295, 5d70adad7130475eab279c5dfb6c0db3; no voters:
I20260928 21:06:30.950475 23692 raft_consensus.cc:2781] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:30.950523 23692 raft_consensus.cc:482] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:30.950549 23692 raft_consensus.cc:3037] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:30.951802 23692 raft_consensus.cc:504] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.952203 23692 leader_election.cc:288] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [CANDIDATE]: Term 1 election: Requested vote from peers 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929), b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67:38165)
I20260928 21:06:30.952595 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "643d6931e687410eb03af075be811692" candidate_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d70adad7130475eab279c5dfb6c0db3"
I20260928 21:06:30.952739 23337 raft_consensus.cc:3037] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:30.952812 23617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "643d6931e687410eb03af075be811692" candidate_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18adfb839fa4409bec450dd9ced5a2f"
I20260928 21:06:30.952927 23617 raft_consensus.cc:3037] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:30.954013 23617 raft_consensus.cc:2454] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1447cd7dbb6b4ab08ec90df4f04a1295 in term 1.
I20260928 21:06:30.954013 23337 raft_consensus.cc:2454] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1447cd7dbb6b4ab08ec90df4f04a1295 in term 1.
I20260928 21:06:30.954310 23410 leader_election.cc:302] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447cd7dbb6b4ab08ec90df4f04a1295, b18adfb839fa4409bec450dd9ced5a2f; no voters:
I20260928 21:06:30.954509 23692 raft_consensus.cc:2781] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:30.954723 23692 raft_consensus.cc:686] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 LEADER]: Becoming Leader. State: Replica: 1447cd7dbb6b4ab08ec90df4f04a1295, State: Running, Role: LEADER
I20260928 21:06:30.955026 23692 consensus_queue.cc:237] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:30.957533 23180 catalog_manager.cc:5495] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66). New cstate: current_term: 1 leader_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:06:31 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:31.016382 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.007814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.21.140.126:43551)
0928 21:06:31.007938 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.007942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.008188 (+ 246us) server_negotiation.cc:407] Connection header received
0928 21:06:31.008292 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.008295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.008337 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.008420 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:31.009071 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.009586 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.010432 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.010643 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.010725 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.013137 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:31.013140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:31.013142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:31.013162 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:31.014510 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:31.015052 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:31.015061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:31.015063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:31.015139 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:31.015432 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:31.015437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:31.015438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:31.015619 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:31.015733 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.016042 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.016245 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":44}
I20260928 21:06:31.016382 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.007774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43551 (local address 127.0.0.1:37754)
0928 21:06:31.008073 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:31.008091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:31.008208 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:31.008454 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:31.008456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:31.008864 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:31.008953 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.008958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.009703 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.009706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.010326 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.010335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.010747 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.010749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.010811 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.011456 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:31.011468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:31.012976 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:31.014621 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:31.014633 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:31.014640 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:31.014886 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:31.015245 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:31.015249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:31.015251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:31.015338 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:31.015733 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:31.015742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:31.015860 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 21:06:31.015990 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 21:06:31.016184 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":128,"threads_started":1}
I20260928 21:06:31.020467 23180 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:37754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:06:31.021272 23180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:31.026665 23317 tablet_service.cc:1467] Processing CreateTablet for tablet 107307932b704ac4b0c5c9064b48fed7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93bed29cf7fb4e2d82310e5ec5c048ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:06:31.026665 23457 tablet_service.cc:1467] Processing CreateTablet for tablet 107307932b704ac4b0c5c9064b48fed7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93bed29cf7fb4e2d82310e5ec5c048ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:06:31.027129 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107307932b704ac4b0c5c9064b48fed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:31.027132 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107307932b704ac4b0c5c9064b48fed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:31.026952 23597 tablet_service.cc:1467] Processing CreateTablet for tablet 107307932b704ac4b0c5c9064b48fed7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93bed29cf7fb4e2d82310e5ec5c048ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:06:31.027652 23597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107307932b704ac4b0c5c9064b48fed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:31.029930 23686 tablet_bootstrap.cc:492] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: Bootstrap starting.
I20260928 21:06:31.030370 23688 tablet_bootstrap.cc:492] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: Bootstrap starting.
I20260928 21:06:31.030758 23687 tablet_bootstrap.cc:492] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: Bootstrap starting.
I20260928 21:06:31.030937 23686 tablet_bootstrap.cc:654] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:31.031334 23688 tablet_bootstrap.cc:654] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:31.031697 23687 tablet_bootstrap.cc:654] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:31.032734 23686 tablet_bootstrap.cc:492] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: No bootstrap required, opened a new log
I20260928 21:06:31.032820 23686 ts_tablet_manager.cc:1397] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:31.032959 23687 tablet_bootstrap.cc:492] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: No bootstrap required, opened a new log
I20260928 21:06:31.033054 23687 ts_tablet_manager.cc:1397] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:06:31.033298 23686 raft_consensus.cc:348] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.033396 23688 tablet_bootstrap.cc:492] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: No bootstrap required, opened a new log
I20260928 21:06:31.033417 23686 raft_consensus.cc:374] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:31.033459 23686 raft_consensus.cc:729] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18adfb839fa4409bec450dd9ced5a2f, State: Initialized, Role: FOLLOWER
I20260928 21:06:31.033474 23688 ts_tablet_manager.cc:1397] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 21:06:31.033514 23687 raft_consensus.cc:348] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.033624 23687 raft_consensus.cc:374] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:31.033604 23686 consensus_queue.cc:260] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.033668 23687 raft_consensus.cc:729] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d70adad7130475eab279c5dfb6c0db3, State: Initialized, Role: FOLLOWER
I20260928 21:06:31.033783 23687 consensus_queue.cc:260] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.033892 23686 ts_tablet_manager.cc:1428] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 21:06:31.033938 23688 raft_consensus.cc:348] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.034092 23688 raft_consensus.cc:374] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:31.034114 23687 ts_tablet_manager.cc:1428] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:31.034126 23688 raft_consensus.cc:729] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1447cd7dbb6b4ab08ec90df4f04a1295, State: Initialized, Role: FOLLOWER
I20260928 21:06:31.034112 23696 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.034240 23696 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.034236 23688 consensus_queue.cc:260] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.034487 23688 ts_tablet_manager.cc:1428] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:31.034471 23694 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.034626 23694 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.034776 23692 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.034912 23692 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.230859 23696 raft_consensus.cc:482] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:31.231025 23696 raft_consensus.cc:504] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.231500 23696 leader_election.cc:288] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929), 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637)
I20260928 21:06:31.232012 23477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107307932b704ac4b0c5c9064b48fed7" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" is_pre_election: true
I20260928 21:06:31.232018 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107307932b704ac4b0c5c9064b48fed7" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d70adad7130475eab279c5dfb6c0db3" is_pre_election: true
I20260928 21:06:31.232160 23477 raft_consensus.cc:2454] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 0.
I20260928 21:06:31.232264 23337 raft_consensus.cc:2454] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 0.
I20260928 21:06:31.232535 23550 leader_election.cc:302] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447cd7dbb6b4ab08ec90df4f04a1295, b18adfb839fa4409bec450dd9ced5a2f; no voters:
I20260928 21:06:31.232801 23696 raft_consensus.cc:2781] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:31.232869 23696 raft_consensus.cc:482] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:31.232898 23696 raft_consensus.cc:3037] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:31.234120 23696 raft_consensus.cc:504] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.234486 23696 leader_election.cc:288] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 election: Requested vote from peers 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929), 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637)
I20260928 21:06:31.235105 23477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107307932b704ac4b0c5c9064b48fed7" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295"
I20260928 21:06:31.235247 23477 raft_consensus.cc:3037] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:31.235234 23337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107307932b704ac4b0c5c9064b48fed7" candidate_uuid: "b18adfb839fa4409bec450dd9ced5a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d70adad7130475eab279c5dfb6c0db3"
I20260928 21:06:31.235390 23337 raft_consensus.cc:3037] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:31.236349 23477 raft_consensus.cc:2454] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 1.
I20260928 21:06:31.236569 23337 raft_consensus.cc:2454] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b18adfb839fa4409bec450dd9ced5a2f in term 1.
I20260928 21:06:31.236645 23550 leader_election.cc:302] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1447cd7dbb6b4ab08ec90df4f04a1295, b18adfb839fa4409bec450dd9ced5a2f; no voters:
I20260928 21:06:31.236800 23696 raft_consensus.cc:2781] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:31.236897 23696 raft_consensus.cc:686] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [term 1 LEADER]: Becoming Leader. State: Replica: b18adfb839fa4409bec450dd9ced5a2f, State: Running, Role: LEADER
I20260928 21:06:31.237032 23696 consensus_queue.cc:237] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } }
I20260928 21:06:31.237702 23693 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: TxnStatusTablet state changed. Reason: New leader b18adfb839fa4409bec450dd9ced5a2f. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.237829 23693 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:06:31.238293 23728 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:06:31.238454 23728 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:06:31.239001 23179 catalog_manager.cc:5495] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f reported cstate change: term changed from 0 to 1, leader changed from <none> to b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67). New cstate: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } health_report { overall_health: HEALTHY } } }
I20260928 21:06:31.273284 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.269501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38165 (local address 127.0.0.1:48202)
0928 21:06:31.269701 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:31.269715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:31.269805 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:31.270081 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:31.270083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:31.270089 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:31.270214 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.270219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.271234 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.271236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.271947 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.271958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.272555 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.272558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.272631 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.272679 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.272987 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.273042 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 21:06:31.273103 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:06:31.273174 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 21:06:31.273319 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.269606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.21.140.67:38165)
0928 21:06:31.269761 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.269766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.269783 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:06:31.269902 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.269905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.269956 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.270056 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:31.270319 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.271107 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.272072 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.272415 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.272521 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.272812 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.272896 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.272967 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.273134 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.273186 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:06:31.275879 23477 raft_consensus.cc:1261] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Refusing update from remote peer b18adfb839fa4409bec450dd9ced5a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:31.275882 23337 raft_consensus.cc:1261] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Refusing update from remote peer b18adfb839fa4409bec450dd9ced5a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:31.276363 23693 consensus_queue.cc:1035] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:31.276546 23696 consensus_queue.cc:1035] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:31.277848 23692 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: TxnStatusTablet state changed. Reason: New leader b18adfb839fa4409bec450dd9ced5a2f. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.278123 23694 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: TxnStatusTablet state changed. Reason: New leader b18adfb839fa4409bec450dd9ced5a2f. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.278249 23694 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.277983 23692 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.279630 23696 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.279750 23696 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:06:31.280370 23699 mvcc.cc:205] Tried to move back new op lower bound from 7334418805861847040 to 7334418805708111872. Current Snapshot: MvccSnapshot[applied={T|T < 7334418805861847040 or (T in {7334418805861847040})}]
I20260928 21:06:31.280956 23693 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.281090 23693 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P b18adfb839fa4409bec450dd9ced5a2f: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:06:31.282339 23694 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.282469 23694 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P 5d70adad7130475eab279c5dfb6c0db3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.282436 23692 tablet_replica.cc:439] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b18adfb839fa4409bec450dd9ced5a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 } } peers { permanent_uuid: "1447cd7dbb6b4ab08ec90df4f04a1295" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44637 } } peers { permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 } } }
I20260928 21:06:31.282537 23692 tablet_replica.cc:442] T 107307932b704ac4b0c5c9064b48fed7 P 1447cd7dbb6b4ab08ec90df4f04a1295: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:31.282614 23700 mvcc.cc:205] Tried to move back new op lower bound from 7334418805861847040 to 7334418805708111872. Current Snapshot: MvccSnapshot[applied={T|T < 7334418805861847040 or (T in {7334418805861847040})}]
I20260928 21:06:31.290733 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.286815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.21.140.66:44637)
0928 21:06:31.287101 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.287109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.287123 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:06:31.287246 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.287249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.287311 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.287432 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:31.287705 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.288230 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.289100 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.289346 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.289452 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.289746 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.290308 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.290440 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.290492 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.290549 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":112,"threads_started":1}
I20260928 21:06:31.302109 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.298313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:50965 (local address 127.21.140.67:38165)
0928 21:06:31.298449 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.298454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.298585 (+ 131us) server_negotiation.cc:407] Connection header received
0928 21:06:31.298742 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.298745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.298792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.298859 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:31.299146 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.299615 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.300878 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.301153 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.301242 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.301597 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.301670 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.301737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.301861 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.301907 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:06:31.302115 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.298172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38165 (local address 127.21.140.66:50965)
0928 21:06:31.298494 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:31.298509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:31.298613 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:31.298892 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:31.298895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:31.298903 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:31.299029 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.299035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.299719 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.299723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.300731 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.300741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.301276 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.301280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.301376 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.301429 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.301764 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.301770 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:06:31.301838 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:06:31.301917 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
I20260928 21:06:31.309499 23337 raft_consensus.cc:1261] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Refusing update from remote peer 1447cd7dbb6b4ab08ec90df4f04a1295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:31.309548 23617 raft_consensus.cc:1261] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Refusing update from remote peer 1447cd7dbb6b4ab08ec90df4f04a1295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:31.310016 23692 consensus_queue.cc:1035] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d70adad7130475eab279c5dfb6c0db3" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:31.310215 23716 consensus_queue.cc:1035] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18adfb839fa4409bec450dd9ced5a2f" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:31.336676 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.332690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:57969 (local address 127.21.140.66:44637)
0928 21:06:31.332798 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.332803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.332963 (+ 160us) server_negotiation.cc:407] Connection header received
0928 21:06:31.333239 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.333241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.333283 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.333349 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:31.333604 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.334395 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.335578 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.335801 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.335885 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.336206 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.336271 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.336373 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.336482 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.336529 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:06:31.336686 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.332519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:44637 (local address 127.21.140.67:57969)
0928 21:06:31.332864 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:31.332879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:31.333053 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:31.333379 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:31.333382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:31.333390 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:31.333499 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.333505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.334493 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.334497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.335461 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:31.335471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.335916 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.335919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:31.336014 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.336078 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:31.336391 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:31.336398 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:06:31.336462 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:06:31.336539 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20260928 21:06:31.363767 23180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:31.363950 23180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:31.366422 23180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f17764e4ae4784aea2de46ddadcf73: Sending DeleteTablet for 3 replicas of tablet 643d6931e687410eb03af075be811692
I20260928 21:06:31.367043 23317 tablet_service.cc:1514] Processing DeleteTablet for tablet 643d6931e687410eb03af075be811692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:31 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:59614
I20260928 21:06:31.367385 23457 tablet_service.cc:1514] Processing DeleteTablet for tablet 643d6931e687410eb03af075be811692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:31 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:47170
I20260928 21:06:31.367973 23597 tablet_service.cc:1514] Processing DeleteTablet for tablet 643d6931e687410eb03af075be811692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:31 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 21:06:31.368435 23747 tablet_replica.cc:330] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: stopping tablet replica
I20260928 21:06:31.368643 23750 tablet_replica.cc:330] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: stopping tablet replica
I20260928 21:06:31.368827 23750 raft_consensus.cc:2229] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:31.369026 23747 raft_consensus.cc:2229] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:31.369236 23750 raft_consensus.cc:2258] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:31.369419 23747 raft_consensus.cc:2258] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:31.370322 23747 ts_tablet_manager.cc:1905] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:31.370323 23750 ts_tablet_manager.cc:1905] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:31.373584 23750 ts_tablet_manager.cc:1918] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:31.373584 23747 ts_tablet_manager.cc:1918] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:31.373682 23750 log.cc:1196] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/643d6931e687410eb03af075be811692
I20260928 21:06:31.373829 23747 log.cc:1196] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/643d6931e687410eb03af075be811692
I20260928 21:06:31.373956 23750 ts_tablet_manager.cc:1939] T 643d6931e687410eb03af075be811692 P b18adfb839fa4409bec450dd9ced5a2f: Deleting consensus metadata
I20260928 21:06:31.374178 23748 tablet_replica.cc:330] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: stopping tablet replica
I20260928 21:06:31.374317 23747 ts_tablet_manager.cc:1939] T 643d6931e687410eb03af075be811692 P 5d70adad7130475eab279c5dfb6c0db3: Deleting consensus metadata
I20260928 21:06:31.374332 23748 raft_consensus.cc:2229] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:31.374686 23748 raft_consensus.cc:2258] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:31.374781 23166 catalog_manager.cc:4841] TS b18adfb839fa4409bec450dd9ced5a2f (127.21.140.67:38165): tablet 643d6931e687410eb03af075be811692 (table test-table [id=c551d31c678841079c0d59943e437eb0]) successfully deleted
I20260928 21:06:31.375213 23165 catalog_manager.cc:4841] TS 5d70adad7130475eab279c5dfb6c0db3 (127.21.140.65:45929): tablet 643d6931e687410eb03af075be811692 (table test-table [id=c551d31c678841079c0d59943e437eb0]) successfully deleted
I20260928 21:06:31.375427 23748 ts_tablet_manager.cc:1905] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:31.378986 23748 ts_tablet_manager.cc:1918] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:31.379084 23748 log.cc:1196] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/643d6931e687410eb03af075be811692
I20260928 21:06:31.379382 23748 ts_tablet_manager.cc:1939] T 643d6931e687410eb03af075be811692 P 1447cd7dbb6b4ab08ec90df4f04a1295: Deleting consensus metadata
I20260928 21:06:31.380331 23166 catalog_manager.cc:4841] TS 1447cd7dbb6b4ab08ec90df4f04a1295 (127.21.140.66:44637): tablet 643d6931e687410eb03af075be811692 (table test-table [id=c551d31c678841079c0d59943e437eb0]) successfully deleted
Sep 28 21:06:31 dist-test-slave-d5wc krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:31.393858 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.380895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.21.140.65:45929)
0928 21:06:31.381607 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.381612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.382009 (+ 397us) server_negotiation.cc:407] Connection header received
0928 21:06:31.382442 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.382446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.382523 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.382610 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:31.383966 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.384597 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.385682 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.385939 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.386100 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.389856 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:31.389870 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:31.389885 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:31.389913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:31.391950 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:31.392429 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:31.392436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:31.392437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:31.392517 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:31.392846 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:31.392851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:31.392853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:31.393159 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:31.393374 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.393547 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.393683 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555}
W20260928 21:06:31.394647 23377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59620
I20260928 21:06:31.404489 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:31.396419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.21.140.126:43551)
0928 21:06:31.396546 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:31.396551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:31.397024 (+ 473us) server_negotiation.cc:407] Connection header received
0928 21:06:31.397157 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:31.397160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:31.397214 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:31.397303 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:31.397949 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.398531 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.399366 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:31.399617 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:31.399732 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:31.401122 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:31.401126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:31.401129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:31.401156 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:31.402717 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:31.403202 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:31.403207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:31.403209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:31.403291 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:31.403623 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:31.403627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:31.403629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:31.403827 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:31.403978 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:31.404201 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 21:06:31.404320 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:06:31.405097 23180 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37762
I20260928 21:06:31.406591 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 23252
I20260928 21:06:31.415832 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 23392
I20260928 21:06:31.422168 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 23532
I20260928 21:06:31.429388 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 23148
2026-09-28T21:06:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3116 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:31 dist-test-slave-d5wc krb5kdc[23756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:31 dist-test-slave-d5wc krb5kdc[23756](info): set up 2 sockets
Sep 28 21:06:31 dist-test-slave-d5wc krb5kdc[23756](info): commencing operation
krb5kdc: starting...
W20260928 21:06:32.463301 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:32 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:32.591193 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:39421
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:32.700948 23769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:32.701238 23769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:32.701278 23769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:32.705382 23769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:32.705442 23769 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:06:32.705458 23769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:32.705476 23769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:32.705518 23769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:32.705554 23769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:32.710773 23769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:39421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:32.711820 23769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:32.712838 23769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:32.721282 23777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:32.721378 23769 server_base.cc:1034] running on GCE node
W20260928 21:06:32.721333 23774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:32.721284 23775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:32.722172 23769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:32.723304 23769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:32.724496 23769 hybrid_clock.cc:648] HybridClock initialized: now 1790629592724485 us; error 27 us; skew 500 ppm
Sep 28 21:06:32 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629592, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:32.727322 23769 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:32.728188 23769 webserver.cc:460] Webserver started at http://127.21.140.126:43533/ using document root <none> and password file <none>
I20260928 21:06:32.729082 23769 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:32.729189 23769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:32.729410 23769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:32.731585 23769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9"
format_stamp: "Formatted at 2026-09-28 21:06:32 on dist-test-slave-d5wc"
server_key: "deb65f771694c528bdaa90e30b0be143"
server_key_iv: "cca8d48d345518cff479287b7b4896ca"
server_key_version: "encryptionkey@0"
I20260928 21:06:32.732177 23769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9"
format_stamp: "Formatted at 2026-09-28 21:06:32 on dist-test-slave-d5wc"
server_key: "deb65f771694c528bdaa90e30b0be143"
server_key_iv: "cca8d48d345518cff479287b7b4896ca"
server_key_version: "encryptionkey@0"
I20260928 21:06:32.736114 23769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 21:06:32.738734 23784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:32.740023 23769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:32.740166 23769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9"
format_stamp: "Formatted at 2026-09-28 21:06:32 on dist-test-slave-d5wc"
server_key: "deb65f771694c528bdaa90e30b0be143"
server_key_iv: "cca8d48d345518cff479287b7b4896ca"
server_key_version: "encryptionkey@0"
I20260928 21:06:32.740280 23769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:32.752264 23769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:32.755400 23769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:32.755609 23769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:32.763649 23769 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:39421
I20260928 21:06:32.763662 23846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:39421 every 8 connection(s)
I20260928 21:06:32.764622 23769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:32.767339 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 23769
I20260928 21:06:32.767470 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:32.767500 23847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:32.767768 22065 external_mini_cluster.cc:1428] Setting key f49c755d3cbeef029780bac92121cb69
I20260928 21:06:32.775143 23847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Bootstrap starting.
Sep 28 21:06:32 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:32.777637 23847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:32.778611 23847 log.cc:824] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:32.781126 23847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: No bootstrap required, opened a new log
I20260928 21:06:32.783846 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:32.769232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.21.140.126:39421)
0928 21:06:32.769670 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:32.769688 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:32.769735 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:06:32.770542 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:32.770566 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:32.771013 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:32.771403 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:32.772058 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:32.773549 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:32.774491 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:32.775186 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:32.775348 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:32.777934 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:32.777941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:32.777979 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:32.778013 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:32.780640 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:32.781102 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:32.781107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:32.781113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:32.781289 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:32.781566 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:32.781569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:32.781570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:32.782117 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:32.782309 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:32.782569 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:06:32.782872 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260928 21:06:32.785171 23847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39421 } }
I20260928 21:06:32.785393 23847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:32.785454 23847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcad6950a3b42e6aecbe1cbfe1103a9, State: Initialized, Role: FOLLOWER
I20260928 21:06:32.785946 23847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39421 } }
I20260928 21:06:32.786146 23847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:32.786214 23847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:32.786316 23847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:32.787269 23847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39421 } }
I20260928 21:06:32.787617 23847 leader_election.cc:302] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bcad6950a3b42e6aecbe1cbfe1103a9; no voters:
I20260928 21:06:32.787983 23847 leader_election.cc:288] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:32.788110 23852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:32.788339 23852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0bcad6950a3b42e6aecbe1cbfe1103a9, State: Running, Role: LEADER
I20260928 21:06:32.788664 23852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39421 } }
I20260928 21:06:32.789232 23847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:32.790328 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39421 } } }
I20260928 21:06:32.790472 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:32.790778 23854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bcad6950a3b42e6aecbe1cbfe1103a9. Latest consensus state: current_term: 1 leader_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bcad6950a3b42e6aecbe1cbfe1103a9" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39421 } } }
I20260928 21:06:32.790868 23854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:32.790953 23861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:32.794872 23861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:32.801124 23861 catalog_manager.cc:1347] Generated new cluster ID: a2d1c47bc6d242288950a4860bf65ba4
I20260928 21:06:32.801206 23861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:32.826164 23861 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:32.827514 23861 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:32.834061 23861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Generated new TSK 0
I20260928 21:06:32.834868 23861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:32.865629 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39421
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:32.977854 23873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:32.978209 23873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:32.978278 23873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:32.982393 23873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:32.982451 23873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:32.982492 23873 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:06:32.982507 23873 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:06:32.982579 23873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:32.987942 23873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:32.989158 23873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:32.990302 23873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:33.000312 23878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:33.000316 23879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:33.000316 23881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:33.000578 23873 server_base.cc:1034] running on GCE node
I20260928 21:06:33.001137 23873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:33.001798 23873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:33.003000 23873 hybrid_clock.cc:648] HybridClock initialized: now 1790629593002978 us; error 35 us; skew 500 ppm
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:33.005936 23873 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:33.007081 23873 webserver.cc:460] Webserver started at http://127.21.140.65:36051/ using document root <none> and password file <none>
I20260928 21:06:33.007670 23873 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:33.007768 23873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:33.008023 23873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:33.010174 23873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "6e58acd871dd45dca8014300c586e1a8"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "2484dea15b4d7abff75751f81c7b5019"
server_key_iv: "61d006be61b3a6cdf31f568b46d68c27"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.010771 23873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "6e58acd871dd45dca8014300c586e1a8"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "2484dea15b4d7abff75751f81c7b5019"
server_key_iv: "61d006be61b3a6cdf31f568b46d68c27"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.014647 23873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:06:33.017115 23888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:33.018448 23873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:33.018585 23873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "6e58acd871dd45dca8014300c586e1a8"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "2484dea15b4d7abff75751f81c7b5019"
server_key_iv: "61d006be61b3a6cdf31f568b46d68c27"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.018704 23873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:33.034583 23873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:33.038197 23873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:33.038414 23873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:33.039997 23873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:33.040084 23873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:33.040154 23873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:33.040203 23873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:33.051710 23873 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:39367
I20260928 21:06:33.051718 24003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:39367 every 8 connection(s)
I20260928 21:06:33.052925 23873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:33.054399 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.040904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:33133 (local address 127.21.140.126:39421)
0928 21:06:33.041112 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.041118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.042001 (+ 883us) server_negotiation.cc:407] Connection header received
0928 21:06:33.043046 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.043050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.043113 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.043179 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.044809 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.045736 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.047108 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.047373 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.047499 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.050059 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.050062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.050064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.050089 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.052074 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.052631 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.052636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.052637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.052746 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.053140 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.053143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.053144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.053329 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.053450 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.054160 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.054277 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:06:33.055183 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.041366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.21.140.65:33133)
0928 21:06:33.041817 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.041867 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.042751 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.043597 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.043603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.043933 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.044625 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.044646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.045885 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.045889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.046966 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.046976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.047548 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.047552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.047731 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.048351 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.048367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.049844 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.052197 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.052202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.052213 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.052471 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.052847 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.052850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.052851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.053035 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.053499 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.053505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.053858 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.054075 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.054329 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
I20260928 21:06:33.056542 24004 heartbeater.cc:347] Connected to a master server at 127.21.140.126:39421
I20260928 21:06:33.056684 24004 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:33.057224 24004 heartbeater.cc:511] Master 127.21.140.126:39421 requested a full tablet report, sending...
I20260928 21:06:33.058876 23800 ts_manager.cc:194] Registered new tserver with Master: 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367)
I20260928 21:06:33.060060 23800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:33133
I20260928 21:06:33.062597 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 23873
I20260928 21:06:33.062747 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:33.063052 22065 external_mini_cluster.cc:1428] Setting key 0eaef48b71675095dd7d7bd236517a33
I20260928 21:06:33.066507 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.058802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.21.140.65:48141)
0928 21:06:33.059081 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.059095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.059216 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.059450 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.059453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.059674 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.059768 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.059773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.060518 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.060522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.061097 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.061104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.061551 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.061554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.061644 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.062208 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.062220 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.062554 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.064747 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.064753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.064759 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.065004 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.065317 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.065320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.065322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.065407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.065819 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.065823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.065916 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.066154 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.066315 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
I20260928 21:06:33.066499 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.058880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:48141 (local address 127.21.140.126:39421)
0928 21:06:33.059044 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.059049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.059173 (+ 124us) server_negotiation.cc:407] Connection header received
0928 21:06:33.059304 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.059314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.059350 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.059434 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.059880 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.060386 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.061216 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.061435 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.061545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.062689 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.062693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.062695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.062718 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.064468 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.065125 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.065130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.065132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.065215 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.065529 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.065537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.065539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.065706 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.065816 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.066212 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.066329 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:33.093194 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39421
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:33.208835 24013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:33.209098 24013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:33.209178 24013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:33.213291 24013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:33.213361 24013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:33.213403 24013 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:06:33.213418 24013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:06:33.213491 24013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:33.218919 24013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:33.220029 24013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:33.221097 24013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:33.231217 24018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:33.231297 24013 server_base.cc:1034] running on GCE node
W20260928 21:06:33.231217 24019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:33.231217 24021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:33.232016 24013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:33.232695 24013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:33.233897 24013 hybrid_clock.cc:648] HybridClock initialized: now 1790629593233873 us; error 38 us; skew 500 ppm
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:33.237140 24013 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:33.238142 24013 webserver.cc:460] Webserver started at http://127.21.140.66:34041/ using document root <none> and password file <none>
I20260928 21:06:33.238736 24013 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:33.238827 24013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:33.239063 24013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:33.241117 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "4ff9170b6db74e85ad9442d91dcc3d08"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "29d2f139fe6ab3c155f9fdf68a574d6e"
server_key_iv: "82d77c763b26a53a99fbba74e6c236bb"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.241750 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "4ff9170b6db74e85ad9442d91dcc3d08"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "29d2f139fe6ab3c155f9fdf68a574d6e"
server_key_iv: "82d77c763b26a53a99fbba74e6c236bb"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.245826 24013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 21:06:33.248386 24028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:33.249811 24013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:33.250020 24013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "4ff9170b6db74e85ad9442d91dcc3d08"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "29d2f139fe6ab3c155f9fdf68a574d6e"
server_key_iv: "82d77c763b26a53a99fbba74e6c236bb"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.250154 24013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:33.273918 24013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:33.277518 24013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:33.277732 24013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:33.279306 24013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:33.279381 24013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:33.279455 24013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:33.279497 24013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:33.291016 24013 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:46371
I20260928 21:06:33.291043 24143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:46371 every 8 connection(s)
I20260928 21:06:33.292119 24013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:33.294010 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.280265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:47981 (local address 127.21.140.126:39421)
0928 21:06:33.280396 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.280400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.281431 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 21:06:33.282574 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.282582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.282656 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.282755 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.284129 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.284781 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.286504 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.286736 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.286858 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.289873 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.289877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.289880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.289912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.291816 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.292432 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.292435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.292437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.292516 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.292934 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.292938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.292940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.293164 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.293245 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.293736 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.293843 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:06:33.294867 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.280777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.21.140.66:47981)
0928 21:06:33.281280 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.281323 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.282276 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.282950 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.282961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.283474 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.283966 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.283980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.284914 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.284920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.286370 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.286380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.286906 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.286909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.287117 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.287738 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.287755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.289679 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.291934 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.291940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.291955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.292316 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.292647 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.292652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.292655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.292819 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.293267 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.293272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.293499 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.293687 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.293942 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":150,"threads_started":1}
I20260928 21:06:33.296046 24144 heartbeater.cc:347] Connected to a master server at 127.21.140.126:39421
I20260928 21:06:33.296216 24144 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:33.296728 24144 heartbeater.cc:511] Master 127.21.140.126:39421 requested a full tablet report, sending...
I20260928 21:06:33.297912 23801 ts_manager.cc:194] Registered new tserver with Master: 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371)
I20260928 21:06:33.298760 23801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:47981
I20260928 21:06:33.300351 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24013
I20260928 21:06:33.300475 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:33.300804 22065 external_mini_cluster.cc:1428] Setting key 03f8db13d44099eb7fd3d7dca07d6744
I20260928 21:06:33.307030 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.298547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.21.140.66:47941)
0928 21:06:33.298861 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.298878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.298990 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.299280 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.299283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.299514 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.299618 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.299624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.300655 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.300668 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.301545 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.301555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.302190 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.302193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.302260 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.302806 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.302818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.303164 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.305332 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.305334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.305336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.305542 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.305846 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.305848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.305850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.305925 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.306515 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.306517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.306620 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.306754 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.306889 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
I20260928 21:06:33.307210 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.298598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:47941 (local address 127.21.140.126:39421)
0928 21:06:33.298716 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.298721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.298947 (+ 226us) server_negotiation.cc:407] Connection header received
0928 21:06:33.299120 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.299124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.299171 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.299266 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.299747 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.300460 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.301701 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.301939 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.302085 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.303334 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.303337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.303340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.303362 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.305217 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.305646 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.305652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.305657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.305754 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.306073 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.306079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.306084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.306413 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.306529 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.306849 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.306997 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:33.330260 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39421
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:33.449201 24153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:33.449553 24153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:33.449674 24153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:33.454679 24153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:33.454769 24153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:33.454849 24153 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:06:33.454891 24153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:06:33.454955 24153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:33.460412 24153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:33.461557 24153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:33.462620 24153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:33.472716 24159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:33.472716 24158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:33.472731 24161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:33.473114 24153 server_base.cc:1034] running on GCE node
I20260928 21:06:33.473507 24153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:33.474188 24153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:33.475380 24153 hybrid_clock.cc:648] HybridClock initialized: now 1790629593475361 us; error 35 us; skew 500 ppm
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:33.478269 24153 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:33.479189 24153 webserver.cc:460] Webserver started at http://127.21.140.67:43677/ using document root <none> and password file <none>
I20260928 21:06:33.479725 24153 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:33.479784 24153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:33.479943 24153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:33.481902 24153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "c6b4acb479f3423d902f2587e063c4b7"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "92bfa0803fb2ad297126e1c1fd9e7bd2"
server_key_iv: "09f07d4e109e1fee1b25dea91b657827"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.482496 24153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "c6b4acb479f3423d902f2587e063c4b7"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "92bfa0803fb2ad297126e1c1fd9e7bd2"
server_key_iv: "09f07d4e109e1fee1b25dea91b657827"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.486191 24153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:33.488692 24168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:33.490442 24153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:33.490547 24153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "c6b4acb479f3423d902f2587e063c4b7"
format_stamp: "Formatted at 2026-09-28 21:06:33 on dist-test-slave-d5wc"
server_key: "92bfa0803fb2ad297126e1c1fd9e7bd2"
server_key_iv: "09f07d4e109e1fee1b25dea91b657827"
server_key_version: "encryptionkey@0"
I20260928 21:06:33.490644 24153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:33.511818 24153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:33.515275 24153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:33.515507 24153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:33.517189 24153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:33.517243 24153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:33.517315 24153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:33.517351 24153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:33.530970 24153 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:46869
I20260928 21:06:33.531944 24153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:33.532155 24283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:46869 every 8 connection(s)
I20260928 21:06:33.533803 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.518205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:42287 (local address 127.21.140.126:39421)
0928 21:06:33.518338 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.518342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.519543 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 21:06:33.520614 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.520618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.520700 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.520765 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.522653 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.523522 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.524803 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.525045 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.525167 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.529546 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.529553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.529559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.529594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.531496 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.532050 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.532055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.532056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.532161 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.532650 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.532653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.532655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.532926 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.533056 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.533561 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.533676 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:06:33.534637 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.518856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.21.140.67:42287)
0928 21:06:33.519365 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.519409 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.520335 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.521253 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.521264 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.521778 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.522442 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.522463 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.523637 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.523640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.524649 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.524658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.525206 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.525210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.525426 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.526100 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.526126 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.529295 (+ 3169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.531627 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.531633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.531649 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.531924 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.532280 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.532283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.532286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.532495 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.533051 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.533056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.533336 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.533494 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.533746 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":128,"threads_started":1}
I20260928 21:06:33.536000 24284 heartbeater.cc:347] Connected to a master server at 127.21.140.126:39421
I20260928 21:06:33.536191 24284 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:33.536773 24284 heartbeater.cc:511] Master 127.21.140.126:39421 requested a full tablet report, sending...
I20260928 21:06:33.537115 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24153
I20260928 21:06:33.537238 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:33.537549 22065 external_mini_cluster.cc:1428] Setting key b8958aaa159887035b0ccbebd7b451f8
I20260928 21:06:33.537794 23800 ts_manager.cc:194] Registered new tserver with Master: c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869)
I20260928 21:06:33.538620 23800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:42287
I20260928 21:06:33.540019 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:06:33.547058 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.538317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:49841 (local address 127.21.140.126:39421)
0928 21:06:33.538453 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.538459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.538775 (+ 316us) server_negotiation.cc:407] Connection header received
0928 21:06:33.538933 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.538936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.538980 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.539065 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.539624 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.540302 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.541672 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.541884 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.542069 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.543227 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.543231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.543233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.543256 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.545180 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.545624 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.545628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.545630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.545724 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.546144 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.546146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.546148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.546318 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.546402 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.546785 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.546900 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:06:33.547245 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.538325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.21.140.67:49841)
0928 21:06:33.538693 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.538707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.538842 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.539100 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.539103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.539313 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.539491 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.539499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.540459 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.540462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.541536 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.541551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.542026 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.542033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.542163 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.542737 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.542748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.543081 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.545301 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.545304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.545305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.545517 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.545852 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.545857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.545860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.545991 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.546427 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.546433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.546574 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.546696 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.546871 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:33.568176 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.559469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.21.140.126:39421)
0928 21:06:33.559624 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.559628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.559640 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:06:33.559677 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.559680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.559715 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.559807 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.560519 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.561059 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.561901 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.562185 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.562296 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.564884 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.564888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.564890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.564915 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.566397 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.566864 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.566868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.566870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.566954 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.567289 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.567294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.567296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.567481 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.567622 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.567869 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.568004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 21:06:33.571360 23800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:33.574419 23800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:33.588555 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.583166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.21.140.66:46371)
0928 21:06:33.584173 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.584185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.584209 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:06:33.584286 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.584291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.584523 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.584702 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.585038 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.585815 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.587269 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.587885 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.587983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.588154 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.588322 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.588412 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":122,"threads_started":1}
I20260928 21:06:33.588553 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.583013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46371 (local address 127.0.0.1:44658)
0928 21:06:33.583705 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.583758 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.583875 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.584700 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.584706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.584726 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.584897 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.584908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.585952 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.585956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.587148 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.587156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.588006 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.588010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.588127 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.588169 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.588279 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.588395 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":87,"threads_started":1}
I20260928 21:06:33.589351 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.583013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:46869 (local address 127.0.0.1:54168)
0928 21:06:33.584365 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.584378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.584455 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.584934 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.584939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.584961 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.585063 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.585069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.586509 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.586512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.587731 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.587741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.588855 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.588859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.588956 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.588978 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.589089 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.589193 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":778,"thread_start_us":107,"threads_started":1}
I20260928 21:06:33.589427 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.583350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:39367 (local address 127.0.0.1:48644)
0928 21:06:33.584092 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.584106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.584195 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.585074 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.585080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.585107 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.585247 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.585254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.586726 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.586729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.587843 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.587853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.588921 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.588925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.589033 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.589071 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.589122 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.589222 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260928 21:06:33.590184 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.583903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.21.140.65:39367)
0928 21:06:33.584687 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.584693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.584712 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:06:33.584777 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.584781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.584935 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.585400 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.585530 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.586605 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.587963 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.588790 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.589695 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.589837 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.589889 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.590016 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":68,"threads_started":1}
I20260928 21:06:33.590595 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.583245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.21.140.67:46869)
0928 21:06:33.584440 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.584448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.584465 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:06:33.584600 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.584606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.584794 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.584945 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.585176 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.586382 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.587864 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.588655 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.588859 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.589012 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.589115 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.590439 (+ 1324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":97,"threads_started":1}
I20260928 21:06:33.592157 24078 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa47f2b6326426bb355b974e95b9026 (DEFAULT_TABLE table=test-table [id=9b9db26fd69f467fa893dfed51e0420c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:33.592639 24218 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa47f2b6326426bb355b974e95b9026 (DEFAULT_TABLE table=test-table [id=9b9db26fd69f467fa893dfed51e0420c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:33.592783 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa47f2b6326426bb355b974e95b9026 (DEFAULT_TABLE table=test-table [id=9b9db26fd69f467fa893dfed51e0420c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:33.593261 24078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa47f2b6326426bb355b974e95b9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:33.593642 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa47f2b6326426bb355b974e95b9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:33.593753 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa47f2b6326426bb355b974e95b9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:33.600286 24307 tablet_bootstrap.cc:492] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Bootstrap starting.
I20260928 21:06:33.601230 24309 tablet_bootstrap.cc:492] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Bootstrap starting.
I20260928 21:06:33.602279 24308 tablet_bootstrap.cc:492] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Bootstrap starting.
I20260928 21:06:33.602577 24307 tablet_bootstrap.cc:654] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:33.603220 24309 tablet_bootstrap.cc:654] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:33.603500 24307 log.cc:824] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:33.603989 24309 log.cc:824] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:33.604315 24308 tablet_bootstrap.cc:654] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:33.605089 24308 log.cc:824] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:33.605559 24307 tablet_bootstrap.cc:492] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: No bootstrap required, opened a new log
I20260928 21:06:33.605767 24307 ts_tablet_manager.cc:1397] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:06:33.605967 24309 tablet_bootstrap.cc:492] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: No bootstrap required, opened a new log
I20260928 21:06:33.606170 24309 ts_tablet_manager.cc:1397] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 21:06:33.606941 24308 tablet_bootstrap.cc:492] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: No bootstrap required, opened a new log
I20260928 21:06:33.607206 24308 ts_tablet_manager.cc:1397] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:06:33.609153 24307 raft_consensus.cc:348] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.609458 24307 raft_consensus.cc:374] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:33.609522 24307 raft_consensus.cc:729] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Initialized, Role: FOLLOWER
I20260928 21:06:33.610015 24307 consensus_queue.cc:260] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.610006 24309 raft_consensus.cc:348] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.610242 24309 raft_consensus.cc:374] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:33.610306 24309 raft_consensus.cc:729] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Initialized, Role: FOLLOWER
I20260928 21:06:33.610713 24284 heartbeater.cc:503] Master 127.21.140.126:39421 was elected leader, sending a full tablet report...
I20260928 21:06:33.611014 24307 ts_tablet_manager.cc:1428] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 21:06:33.611299 24309 consensus_queue.cc:260] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.611557 24308 raft_consensus.cc:348] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.611804 24308 raft_consensus.cc:374] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:33.611855 24308 raft_consensus.cc:729] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e58acd871dd45dca8014300c586e1a8, State: Initialized, Role: FOLLOWER
I20260928 21:06:33.612285 24309 ts_tablet_manager.cc:1428] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260928 21:06:33.612327 24144 heartbeater.cc:503] Master 127.21.140.126:39421 was elected leader, sending a full tablet report...
I20260928 21:06:33.612286 24308 consensus_queue.cc:260] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.612859 24004 heartbeater.cc:503] Master 127.21.140.126:39421 was elected leader, sending a full tablet report...
I20260928 21:06:33.613178 24308 ts_tablet_manager.cc:1428] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:06:33.713541 24313 raft_consensus.cc:482] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:33.713743 24313 raft_consensus.cc:504] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.714897 24313 leader_election.cc:288] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371), 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367)
I20260928 21:06:33.719411 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.715277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:38675 (local address 127.21.140.66:46371)
0928 21:06:33.715395 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.715401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.715545 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:06:33.715620 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.715623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.715691 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.715773 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.716135 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.716927 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.718200 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.718835 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.718966 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.719033 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.719217 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.719286 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:06:33.719411 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.715196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46371 (local address 127.21.140.67:38675)
0928 21:06:33.715393 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.715409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.715512 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.715801 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.715805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.715823 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.715954 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.715961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.717041 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.717045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.718064 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.718073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.718961 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.718964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.719087 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.719106 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.719201 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.719276 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 21:06:33.720052 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.715359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:48683 (local address 127.21.140.65:39367)
0928 21:06:33.715508 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.715512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.715725 (+ 213us) server_negotiation.cc:407] Connection header received
0928 21:06:33.715830 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.715833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.715877 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.715961 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.716781 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.717456 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.718743 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.719301 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.719445 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.719492 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.719871 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.719945 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:06:33.720043 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.715241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:39367 (local address 127.21.140.67:48683)
0928 21:06:33.715620 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.715634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.715743 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.716087 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.716094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.716116 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.716650 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.716660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.717597 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.717600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.718610 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.718621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.719444 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.719448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.719729 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.719756 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.719840 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.719908 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":144,"threads_started":1}
I20260928 21:06:33.720255 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa47f2b6326426bb355b974e95b9026" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" is_pre_election: true
I20260928 21:06:33.720625 24098 raft_consensus.cc:2454] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 0.
I20260928 21:06:33.720821 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa47f2b6326426bb355b974e95b9026" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8" is_pre_election: true
I20260928 21:06:33.721097 23958 raft_consensus.cc:2454] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 0.
I20260928 21:06:33.721148 24171 leader_election.cc:302] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:33.721411 24313 raft_consensus.cc:2781] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:33.721526 24313 raft_consensus.cc:482] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:33.721612 24313 raft_consensus.cc:3037] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:33.722761 24313 raft_consensus.cc:504] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.723203 24313 leader_election.cc:288] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371), 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367)
I20260928 21:06:33.723521 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa47f2b6326426bb355b974e95b9026" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9170b6db74e85ad9442d91dcc3d08"
I20260928 21:06:33.723595 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa47f2b6326426bb355b974e95b9026" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8"
I20260928 21:06:33.723666 24098 raft_consensus.cc:3037] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:33.723767 23958 raft_consensus.cc:3037] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:33.724751 24098 raft_consensus.cc:2454] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 1.
I20260928 21:06:33.724854 23958 raft_consensus.cc:2454] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 1.
I20260928 21:06:33.725090 24171 leader_election.cc:302] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:33.725318 24313 raft_consensus.cc:2781] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:33.725564 24313 raft_consensus.cc:686] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 1 LEADER]: Becoming Leader. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Running, Role: LEADER
I20260928 21:06:33.725893 24313 consensus_queue.cc:237] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } }
I20260928 21:06:33.728976 23800 catalog_manager.cc:5495] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "c6b4acb479f3423d902f2587e063c4b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:06:33 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629592, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:33.764451 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.755795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39421 (local address 127.0.0.1:42196)
0928 21:06:33.756136 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.756152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.756303 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.756540 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.756543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.756978 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:33.757143 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.757170 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.757951 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.757955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.758756 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.758766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.759177 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.759179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.759243 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.759872 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:33.759884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:33.761320 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:33.762876 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.762886 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.762891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.763125 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.763450 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:33.763455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:33.763457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:33.763560 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:33.763974 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:33.763979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:33.764081 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.764191 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.764318 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260928 21:06:33.764564 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.755905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.21.140.126:39421)
0928 21:06:33.756075 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.756080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.756237 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:06:33.756394 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.756396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.756435 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.756504 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:33.757272 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.757804 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.758860 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.759068 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.759148 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.761461 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:33.761464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:33.761466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:33.761486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:33.762777 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.763230 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.763233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.763234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.763341 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:33.763653 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:33.763657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:33.763659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:33.763872 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:33.763971 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.764272 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.764395 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 21:06:33.768885 23800 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:42196:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:06:33.769657 23800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:33.774858 24078 tablet_service.cc:1467] Processing CreateTablet for tablet 6380ea4574a04e4aac7a6ad67898961f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18cb1cd192b3435dbcf6911819adad26]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:06:33.774791 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 6380ea4574a04e4aac7a6ad67898961f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18cb1cd192b3435dbcf6911819adad26]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:06:33.774820 24218 tablet_service.cc:1467] Processing CreateTablet for tablet 6380ea4574a04e4aac7a6ad67898961f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18cb1cd192b3435dbcf6911819adad26]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:06:33.775578 24078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6380ea4574a04e4aac7a6ad67898961f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:33.775580 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6380ea4574a04e4aac7a6ad67898961f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:33.775580 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6380ea4574a04e4aac7a6ad67898961f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:33.778749 24307 tablet_bootstrap.cc:492] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: Bootstrap starting.
I20260928 21:06:33.778775 24308 tablet_bootstrap.cc:492] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: Bootstrap starting.
I20260928 21:06:33.779619 24308 tablet_bootstrap.cc:654] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:33.779623 24307 tablet_bootstrap.cc:654] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:33.780121 24309 tablet_bootstrap.cc:492] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: Bootstrap starting.
I20260928 21:06:33.781191 24307 tablet_bootstrap.cc:492] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: No bootstrap required, opened a new log
I20260928 21:06:33.781203 24309 tablet_bootstrap.cc:654] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:33.781279 24307 ts_tablet_manager.cc:1397] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 21:06:33.781572 24308 tablet_bootstrap.cc:492] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: No bootstrap required, opened a new log
I20260928 21:06:33.781631 24308 ts_tablet_manager.cc:1397] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:33.781769 24307 raft_consensus.cc:348] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.781867 24307 raft_consensus.cc:374] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:33.781889 24307 raft_consensus.cc:729] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Initialized, Role: FOLLOWER
I20260928 21:06:33.782078 24307 consensus_queue.cc:260] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.782168 24308 raft_consensus.cc:348] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.782292 24308 raft_consensus.cc:374] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:33.782326 24308 raft_consensus.cc:729] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e58acd871dd45dca8014300c586e1a8, State: Initialized, Role: FOLLOWER
I20260928 21:06:33.782369 24307 ts_tablet_manager.cc:1428] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:33.782434 24308 consensus_queue.cc:260] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.782465 24309 tablet_bootstrap.cc:492] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: No bootstrap required, opened a new log
I20260928 21:06:33.782373 24313 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.782552 24309 ts_tablet_manager.cc:1397] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 21:06:33.782608 24313 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:33.782665 24308 ts_tablet_manager.cc:1428] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:33.782951 24315 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.783085 24315 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:33.783036 24309 raft_consensus.cc:348] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.783159 24309 raft_consensus.cc:374] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:33.783200 24309 raft_consensus.cc:729] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Initialized, Role: FOLLOWER
I20260928 21:06:33.783322 24309 consensus_queue.cc:260] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
W20260928 21:06:33.783604 24285 tablet.cc:2369] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:33.784325 24309 ts_tablet_manager.cc:1428] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:06:33.784284 24314 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.784418 24314 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 21:06:33.793325 24145 tablet.cc:2369] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:06:33.804508 24005 tablet.cc:2369] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:33.870285 24314 raft_consensus.cc:482] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:33.870440 24314 raft_consensus.cc:504] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.871569 24314 leader_election.cc:288] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869)
I20260928 21:06:33.876011 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.871911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:46869 (local address 127.21.140.66:37343)
0928 21:06:33.872447 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.872460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.872596 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.872848 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.872851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.872867 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.872973 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.872980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.873695 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.873698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.874737 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.874767 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.875564 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.875567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.875663 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.875690 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.875765 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.875820 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":290,"thread_start_us":77,"threads_started":1}
I20260928 21:06:33.876047 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.872044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:37343 (local address 127.21.140.67:46869)
0928 21:06:33.872182 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.872188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.872547 (+ 359us) server_negotiation.cc:407] Connection header received
0928 21:06:33.872698 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.872701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.872747 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.872836 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.873077 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.873578 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.874901 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.875422 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.875557 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.875618 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.875800 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.875898 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:06:33.876113 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.872105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:51379 (local address 127.21.140.65:39367)
0928 21:06:33.872256 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.872260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.872312 (+ 52us) server_negotiation.cc:407] Connection header received
0928 21:06:33.872493 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.872497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.872554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.872662 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:33.872965 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.873520 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.875016 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.875446 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.875598 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.875651 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.875899 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.875982 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:06:33.876655 24238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380ea4574a04e4aac7a6ad67898961f" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b4acb479f3423d902f2587e063c4b7" is_pre_election: true
I20260928 21:06:33.876796 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.871911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:39367 (local address 127.21.140.66:51379)
0928 21:06:33.872226 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.872241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.872349 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.872686 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.872690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.872708 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:33.872838 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.872843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.873642 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.873649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.874892 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.874902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.875605 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.875609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.875731 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.875783 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.875866 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.875934 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":95,"threads_started":1}
I20260928 21:06:33.876912 24238 raft_consensus.cc:2454] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 0.
I20260928 21:06:33.877358 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380ea4574a04e4aac7a6ad67898961f" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8" is_pre_election: true
I20260928 21:06:33.877417 24030 leader_election.cc:302] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:33.877517 23958 raft_consensus.cc:2454] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 0.
I20260928 21:06:33.877739 24314 raft_consensus.cc:2781] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:33.877820 24314 raft_consensus.cc:482] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:33.877879 24314 raft_consensus.cc:3037] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:33.879000 24314 raft_consensus.cc:504] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.879365 24314 leader_election.cc:288] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 election: Requested vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869)
I20260928 21:06:33.879766 24238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380ea4574a04e4aac7a6ad67898961f" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b4acb479f3423d902f2587e063c4b7"
I20260928 21:06:33.879769 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380ea4574a04e4aac7a6ad67898961f" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8"
I20260928 21:06:33.879889 23958 raft_consensus.cc:3037] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:33.879899 24238 raft_consensus.cc:3037] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:33.880813 24238 raft_consensus.cc:2454] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 1.
I20260928 21:06:33.880813 23958 raft_consensus.cc:2454] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 1.
I20260928 21:06:33.881111 24030 leader_election.cc:302] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:33.881301 24314 raft_consensus.cc:2781] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:33.881531 24314 raft_consensus.cc:686] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Running, Role: LEADER
I20260928 21:06:33.881898 24314 consensus_queue.cc:237] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:33.884219 24330 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: TxnStatusTablet state changed. Reason: New leader 4ff9170b6db74e85ad9442d91dcc3d08. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.884352 24330 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:06:33.884665 24332 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:06:33.884872 24332 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:06:33.885605 23801 catalog_manager.cc:5495] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66). New cstate: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } health_report { overall_health: UNKNOWN } } }
I20260928 21:06:33.908679 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.904814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46371 (local address 127.0.0.1:44672)
0928 21:06:33.904976 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.904994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.905105 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.905454 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.905457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.905465 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:33.905583 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.905589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.906358 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.906362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.907010 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.907019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.907511 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.907515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.907609 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.907658 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.908364 (+ 706us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.908384 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.908456 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.908557 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 21:06:33.908679 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.904959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.21.140.66:46371)
0928 21:06:33.905100 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.905104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.905119 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:06:33.905265 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.905268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.905335 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.905439 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:33.905686 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.906239 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.907138 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.907378 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.907489 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.907807 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.908257 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.908388 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.908467 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.908519 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:06:33.914270 24238 raft_consensus.cc:1261] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Refusing update from remote peer 4ff9170b6db74e85ad9442d91dcc3d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:33.914330 23958 raft_consensus.cc:1261] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Refusing update from remote peer 4ff9170b6db74e85ad9442d91dcc3d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:33.914916 24330 consensus_queue.cc:1035] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:33.915179 24314 consensus_queue.cc:1035] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:33.918597 24313 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: TxnStatusTablet state changed. Reason: New leader 4ff9170b6db74e85ad9442d91dcc3d08. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.918731 24313 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:33.918856 24315 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: TxnStatusTablet state changed. Reason: New leader 4ff9170b6db74e85ad9442d91dcc3d08. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.918979 24315 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:33.921423 24330 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.921550 24330 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:06:33.921677 24330 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.921782 24330 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 4ff9170b6db74e85ad9442d91dcc3d08: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:06:33.921892 24335 mvcc.cc:205] Tried to move back new op lower bound from 7334418816665448448 to 7334418816543268864. Current Snapshot: MvccSnapshot[applied={T|T < 7334418816665448448}]
I20260928 21:06:33.922729 24315 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.922847 24315 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P 6e58acd871dd45dca8014300c586e1a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:33.923352 24313 tablet_replica.cc:439] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } } }
I20260928 21:06:33.923462 24313 tablet_replica.cc:442] T 6380ea4574a04e4aac7a6ad67898961f P c6b4acb479f3423d902f2587e063c4b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:06:33.927086 24336 mvcc.cc:205] Tried to move back new op lower bound from 7334418816665448448 to 7334418816543268864. Current Snapshot: MvccSnapshot[applied={T|T < 7334418816665448448}]
I20260928 21:06:33.928996 24337 mvcc.cc:205] Tried to move back new op lower bound from 7334418816665448448 to 7334418816543268864. Current Snapshot: MvccSnapshot[applied={T|T < 7334418816665448448}]
I20260928 21:06:33.933180 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.929082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.21.140.67:46869)
0928 21:06:33.929223 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.929228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.929319 (+ 91us) server_negotiation.cc:407] Connection header received
0928 21:06:33.929512 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.929515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.929577 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.929672 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:33.930070 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.930778 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.931731 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.931967 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.932148 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.932420 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.932803 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.932926 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.932996 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.933046 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:06:33.944231 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.940687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:35133 (local address 127.21.140.66:46371)
0928 21:06:33.940822 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.940826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.940839 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:06:33.940959 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.940973 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.941024 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.941088 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:33.941404 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.941891 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.943150 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.943360 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.943487 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.943828 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.943898 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.943956 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.944073 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.944116 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:06:33.944281 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.940529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46371 (local address 127.21.140.67:35133)
0928 21:06:33.940711 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.940727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.940813 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.941154 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.941157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.941165 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:33.941288 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.941294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.942023 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.942027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.943010 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.943020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.943497 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.943501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.943600 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.943660 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.943987 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.943994 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.944063 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.944144 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 21:06:33.951413 23958 raft_consensus.cc:1261] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Refusing update from remote peer c6b4acb479f3423d902f2587e063c4b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:33.951718 24098 raft_consensus.cc:1261] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Refusing update from remote peer c6b4acb479f3423d902f2587e063c4b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:33.951915 24313 consensus_queue.cc:1035] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:33.952297 24317 consensus_queue.cc:1035] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:33.977387 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.973244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:34415 (local address 127.21.140.67:46869)
0928 21:06:33.973358 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:33.973362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:33.973750 (+ 388us) server_negotiation.cc:407] Connection header received
0928 21:06:33.973865 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:33.973868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:33.973917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:33.974043 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:33.974295 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.975084 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.976188 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.976431 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.976580 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.976884 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.976972 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.977044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:33.977184 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:06:33.977244 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:06:33.977849 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:33.973226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:46869 (local address 127.21.140.66:34415)
0928 21:06:33.973644 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:33.973659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:33.973772 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:33.974053 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:33.974056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:33.974063 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:33.974187 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.974194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.975203 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.975206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.976066 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:33.976078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:33.976577 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:33.976580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:33.976680 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:33.976733 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:33.977082 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:33.977089 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:06:33.977540 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 21:06:33.977634 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":165,"threads_started":1}
I20260928 21:06:34.002225 23801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:34.002439 23801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:34.005312 23801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Sending DeleteTablet for 3 replicas of tablet 3fa47f2b6326426bb355b974e95b9026
I20260928 21:06:34.006350 24218 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa47f2b6326426bb355b974e95b9026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:33 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:54168
I20260928 21:06:34.006453 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa47f2b6326426bb355b974e95b9026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:33 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:48644
I20260928 21:06:34.007007 24078 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa47f2b6326426bb355b974e95b9026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:33 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:44658
I20260928 21:06:34.007259 24354 tablet_replica.cc:330] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: stopping tablet replica
I20260928 21:06:34.007658 24354 raft_consensus.cc:2229] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:34.008075 24354 raft_consensus.cc:2258] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:34.008318 23801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:34.008752 23801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:34.009653 24355 tablet_replica.cc:330] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: stopping tablet replica
I20260928 21:06:34.009955 24355 raft_consensus.cc:2229] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:34.010213 24355 raft_consensus.cc:2258] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:34.010622 24356 tablet_replica.cc:330] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: stopping tablet replica
I20260928 21:06:34.010885 24356 raft_consensus.cc:2229] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:34.011037 24356 raft_consensus.cc:2258] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:34.012293 24355 ts_tablet_manager.cc:1905] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:34.015352 24355 ts_tablet_manager.cc:1918] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:34.015453 24355 log.cc:1196] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/3fa47f2b6326426bb355b974e95b9026
I20260928 21:06:34.015769 24355 ts_tablet_manager.cc:1939] T 3fa47f2b6326426bb355b974e95b9026 P 6e58acd871dd45dca8014300c586e1a8: Deleting consensus metadata
I20260928 21:06:34.017040 23785 catalog_manager.cc:4841] TS 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367): tablet 3fa47f2b6326426bb355b974e95b9026 (table test-table [id=9b9db26fd69f467fa893dfed51e0420c]) successfully deleted
I20260928 21:06:34.017620 23938 tablet_service.cc:1467] Processing CreateTablet for tablet f32b4f812e764e25a780e88f72ac2404 (DEFAULT_TABLE table=test-table [id=9b6e10bf86654b85a60a012612b9ef2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:34.017920 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32b4f812e764e25a780e88f72ac2404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:34.018083 24218 tablet_service.cc:1467] Processing CreateTablet for tablet f32b4f812e764e25a780e88f72ac2404 (DEFAULT_TABLE table=test-table [id=9b6e10bf86654b85a60a012612b9ef2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:34.018442 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32b4f812e764e25a780e88f72ac2404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:34.020325 24308 tablet_bootstrap.cc:492] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Bootstrap starting.
I20260928 21:06:34.021656 24308 tablet_bootstrap.cc:654] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:34.021709 24354 ts_tablet_manager.cc:1905] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:34.021831 24078 tablet_service.cc:1467] Processing CreateTablet for tablet f32b4f812e764e25a780e88f72ac2404 (DEFAULT_TABLE table=test-table [id=9b6e10bf86654b85a60a012612b9ef2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:34.022094 24078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32b4f812e764e25a780e88f72ac2404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:34.024012 24356 ts_tablet_manager.cc:1905] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:34.024480 24309 tablet_bootstrap.cc:492] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Bootstrap starting.
I20260928 21:06:34.025408 24309 tablet_bootstrap.cc:654] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:34.025816 24307 tablet_bootstrap.cc:492] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Bootstrap starting.
I20260928 21:06:34.026279 24354 ts_tablet_manager.cc:1918] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:34.026403 24354 log.cc:1196] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/3fa47f2b6326426bb355b974e95b9026
I20260928 21:06:34.026744 24354 ts_tablet_manager.cc:1939] T 3fa47f2b6326426bb355b974e95b9026 P c6b4acb479f3423d902f2587e063c4b7: Deleting consensus metadata
I20260928 21:06:34.026860 24356 ts_tablet_manager.cc:1918] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:34.026945 24356 log.cc:1196] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/3fa47f2b6326426bb355b974e95b9026
I20260928 21:06:34.026963 24307 tablet_bootstrap.cc:654] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:34.027724 24356 ts_tablet_manager.cc:1939] T 3fa47f2b6326426bb355b974e95b9026 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting consensus metadata
I20260928 21:06:34.027940 23786 catalog_manager.cc:4841] TS c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869): tablet 3fa47f2b6326426bb355b974e95b9026 (table test-table [id=9b9db26fd69f467fa893dfed51e0420c]) successfully deleted
I20260928 21:06:34.028985 23787 catalog_manager.cc:4841] TS 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371): tablet 3fa47f2b6326426bb355b974e95b9026 (table test-table [id=9b9db26fd69f467fa893dfed51e0420c]) successfully deleted
I20260928 21:06:34.029874 24307 tablet_bootstrap.cc:492] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: No bootstrap required, opened a new log
I20260928 21:06:34.030042 24307 ts_tablet_manager.cc:1397] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:06:34.030143 24309 tablet_bootstrap.cc:492] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: No bootstrap required, opened a new log
I20260928 21:06:34.030225 24309 ts_tablet_manager.cc:1397] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260928 21:06:34.030387 24308 tablet_bootstrap.cc:492] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: No bootstrap required, opened a new log
I20260928 21:06:34.030499 24308 ts_tablet_manager.cc:1397] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 21:06:34.030558 24307 raft_consensus.cc:348] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.030606 24309 raft_consensus.cc:348] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.030678 24307 raft_consensus.cc:374] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:34.030694 24309 raft_consensus.cc:374] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:34.030714 24307 raft_consensus.cc:729] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Initialized, Role: FOLLOWER
I20260928 21:06:34.030731 24309 raft_consensus.cc:729] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Initialized, Role: FOLLOWER
I20260928 21:06:34.030848 24307 consensus_queue.cc:260] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.030848 24309 consensus_queue.cc:260] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.030957 24308 raft_consensus.cc:348] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.031077 24308 raft_consensus.cc:374] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:34.031098 24307 ts_tablet_manager.cc:1428] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:34.031108 24309 ts_tablet_manager.cc:1428] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:34.031107 24308 raft_consensus.cc:729] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e58acd871dd45dca8014300c586e1a8, State: Initialized, Role: FOLLOWER
I20260928 21:06:34.031251 24308 consensus_queue.cc:260] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.031523 24308 ts_tablet_manager.cc:1428] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:34.412767 24313 raft_consensus.cc:482] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:34.412981 24313 raft_consensus.cc:504] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.413532 24313 leader_election.cc:288] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371)
I20260928 21:06:34.414068 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32b4f812e764e25a780e88f72ac2404" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8" is_pre_election: true
I20260928 21:06:34.414234 23958 raft_consensus.cc:2454] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 0.
I20260928 21:06:34.414153 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32b4f812e764e25a780e88f72ac2404" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" is_pre_election: true
I20260928 21:06:34.414359 24098 raft_consensus.cc:2454] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 0.
I20260928 21:06:34.414602 24169 leader_election.cc:302] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e58acd871dd45dca8014300c586e1a8, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:34.414815 24313 raft_consensus.cc:2781] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:34.414891 24313 raft_consensus.cc:482] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:34.414937 24313 raft_consensus.cc:3037] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:34.415915 24313 raft_consensus.cc:504] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.416253 24313 leader_election.cc:288] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371)
I20260928 21:06:34.416615 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32b4f812e764e25a780e88f72ac2404" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9170b6db74e85ad9442d91dcc3d08"
I20260928 21:06:34.416654 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32b4f812e764e25a780e88f72ac2404" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8"
I20260928 21:06:34.416759 24098 raft_consensus.cc:3037] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:34.416790 23958 raft_consensus.cc:3037] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:34.417682 24098 raft_consensus.cc:2454] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 1.
I20260928 21:06:34.417711 23958 raft_consensus.cc:2454] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 1.
I20260928 21:06:34.417996 24171 leader_election.cc:302] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:34.418193 24313 raft_consensus.cc:2781] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:34.418319 24313 raft_consensus.cc:686] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 1 LEADER]: Becoming Leader. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Running, Role: LEADER
I20260928 21:06:34.418468 24313 consensus_queue.cc:237] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:34.420132 23799 catalog_manager.cc:5495] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "c6b4acb479f3423d902f2587e063c4b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } health_report { overall_health: HEALTHY } } }
I20260928 21:06:34.493598 24098 raft_consensus.cc:1261] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Refusing update from remote peer c6b4acb479f3423d902f2587e063c4b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:34.493662 23958 raft_consensus.cc:1261] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Refusing update from remote peer c6b4acb479f3423d902f2587e063c4b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:34.494028 24313 consensus_queue.cc:1035] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:34.494204 24317 consensus_queue.cc:1035] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:06:34.508915 24337 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:06:36.075117 23800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:36.075403 23800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:36.079286 23800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Sending DeleteTablet for 3 replicas of tablet f32b4f812e764e25a780e88f72ac2404
I20260928 21:06:36.080296 24218 tablet_service.cc:1514] Processing DeleteTablet for tablet f32b4f812e764e25a780e88f72ac2404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:54168
I20260928 21:06:36.080299 24071 tablet_service.cc:1514] Processing DeleteTablet for tablet f32b4f812e764e25a780e88f72ac2404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:44658
I20260928 21:06:36.080355 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet f32b4f812e764e25a780e88f72ac2404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:48644
I20260928 21:06:36.080816 24388 tablet_replica.cc:330] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: stopping tablet replica
I20260928 21:06:36.080818 24389 tablet_replica.cc:330] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: stopping tablet replica
I20260928 21:06:36.081013 24387 tablet_replica.cc:330] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: stopping tablet replica
I20260928 21:06:36.081048 24389 raft_consensus.cc:2229] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:36.081141 24388 raft_consensus.cc:2229] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:36.081216 24387 raft_consensus.cc:2229] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:36.081346 24389 raft_consensus.cc:2258] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:36.081393 24387 raft_consensus.cc:2258] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:36.081540 24388 raft_consensus.cc:2258] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:36.082376 24387 ts_tablet_manager.cc:1905] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:36.082376 24388 ts_tablet_manager.cc:1905] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:36.082376 24389 ts_tablet_manager.cc:1905] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:36.085988 24387 ts_tablet_manager.cc:1918] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:06:36.085958 24388 ts_tablet_manager.cc:1918] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:06:36.086171 24387 log.cc:1196] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/f32b4f812e764e25a780e88f72ac2404
I20260928 21:06:36.086107 24388 log.cc:1196] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/f32b4f812e764e25a780e88f72ac2404
I20260928 21:06:36.086565 24387 ts_tablet_manager.cc:1939] T f32b4f812e764e25a780e88f72ac2404 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting consensus metadata
I20260928 21:06:36.086622 24388 ts_tablet_manager.cc:1939] T f32b4f812e764e25a780e88f72ac2404 P c6b4acb479f3423d902f2587e063c4b7: Deleting consensus metadata
I20260928 21:06:36.087074 24389 ts_tablet_manager.cc:1918] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:06:36.087266 24389 log.cc:1196] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/f32b4f812e764e25a780e88f72ac2404
I20260928 21:06:36.087679 24389 ts_tablet_manager.cc:1939] T f32b4f812e764e25a780e88f72ac2404 P 6e58acd871dd45dca8014300c586e1a8: Deleting consensus metadata
I20260928 21:06:36.087771 23786 catalog_manager.cc:4841] TS c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869): tablet f32b4f812e764e25a780e88f72ac2404 (table test-table [id=9b6e10bf86654b85a60a012612b9ef2f]) successfully deleted
I20260928 21:06:36.087771 23787 catalog_manager.cc:4841] TS 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371): tablet f32b4f812e764e25a780e88f72ac2404 (table test-table [id=9b6e10bf86654b85a60a012612b9ef2f]) successfully deleted
I20260928 21:06:36.088866 23785 catalog_manager.cc:4841] TS 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367): tablet f32b4f812e764e25a780e88f72ac2404 (table test-table [id=9b6e10bf86654b85a60a012612b9ef2f]) successfully deleted
Sep 28 21:06:36 dist-test-slave-d5wc krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:06:36 dist-test-slave-d5wc krb5kdc[23756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:36.126024 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:36.106870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.21.140.126:39421)
0928 21:06:36.107259 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:36.107272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:36.107375 (+ 103us) server_negotiation.cc:407] Connection header received
0928 21:06:36.107619 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:36.107623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:36.107680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:36.107795 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:36.108726 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.109363 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.110516 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.110802 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.110939 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:36.121537 (+ 10598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:36.121542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:36.121545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:36.121599 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:36.123886 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:36.124537 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:36.124540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:36.124542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:36.124641 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:36.124991 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:36.124993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:36.124995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:36.125232 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:36.125354 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:36.125632 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 21:06:36.125796 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":142,"threads_started":1}
I20260928 21:06:36.129669 23800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:36.130393 23800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:36.137040 23938 tablet_service.cc:1467] Processing CreateTablet for tablet 89434938dadc4762b7131a9aa2852f46 (DEFAULT_TABLE table=test-table [id=0c2f53ddf66448bcad902e6a82c8ffa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:36.137055 24071 tablet_service.cc:1467] Processing CreateTablet for tablet 89434938dadc4762b7131a9aa2852f46 (DEFAULT_TABLE table=test-table [id=0c2f53ddf66448bcad902e6a82c8ffa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:36.137205 24218 tablet_service.cc:1467] Processing CreateTablet for tablet 89434938dadc4762b7131a9aa2852f46 (DEFAULT_TABLE table=test-table [id=0c2f53ddf66448bcad902e6a82c8ffa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:36.137513 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89434938dadc4762b7131a9aa2852f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:36.137512 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89434938dadc4762b7131a9aa2852f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:36.137524 24071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89434938dadc4762b7131a9aa2852f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:36.141144 24400 tablet_bootstrap.cc:492] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Bootstrap starting.
I20260928 21:06:36.141330 24399 tablet_bootstrap.cc:492] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Bootstrap starting.
I20260928 21:06:36.141412 24398 tablet_bootstrap.cc:492] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Bootstrap starting.
I20260928 21:06:36.142496 24398 tablet_bootstrap.cc:654] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:36.142503 24400 tablet_bootstrap.cc:654] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:36.142899 24399 tablet_bootstrap.cc:654] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:36.144158 24400 tablet_bootstrap.cc:492] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: No bootstrap required, opened a new log
I20260928 21:06:36.144260 24398 tablet_bootstrap.cc:492] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: No bootstrap required, opened a new log
I20260928 21:06:36.144289 24400 ts_tablet_manager.cc:1397] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:36.144369 24398 ts_tablet_manager.cc:1397] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:36.144472 24399 tablet_bootstrap.cc:492] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: No bootstrap required, opened a new log
I20260928 21:06:36.144565 24399 ts_tablet_manager.cc:1397] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:36.144887 24400 raft_consensus.cc:348] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.144870 24398 raft_consensus.cc:348] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.145030 24398 raft_consensus.cc:374] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:36.145025 24400 raft_consensus.cc:374] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:36.145058 24398 raft_consensus.cc:729] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Initialized, Role: FOLLOWER
I20260928 21:06:36.145062 24400 raft_consensus.cc:729] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Initialized, Role: FOLLOWER
I20260928 21:06:36.145066 24399 raft_consensus.cc:348] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.145184 24399 raft_consensus.cc:374] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:36.145221 24399 raft_consensus.cc:729] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e58acd871dd45dca8014300c586e1a8, State: Initialized, Role: FOLLOWER
I20260928 21:06:36.145216 24398 consensus_queue.cc:260] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.145215 24400 consensus_queue.cc:260] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.145376 24399 consensus_queue.cc:260] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.145499 24398 ts_tablet_manager.cc:1428] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:06:36.145517 24400 ts_tablet_manager.cc:1428] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 21:06:36.145793 24399 ts_tablet_manager.cc:1428] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 21:06:36.204877 24364 raft_consensus.cc:482] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:36.205127 24364 raft_consensus.cc:504] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.205832 24364 leader_election.cc:288] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371)
I20260928 21:06:36.206516 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89434938dadc4762b7131a9aa2852f46" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" is_pre_election: true
I20260928 21:06:36.206517 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89434938dadc4762b7131a9aa2852f46" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8" is_pre_election: true
I20260928 21:06:36.206750 23958 raft_consensus.cc:2454] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 0.
I20260928 21:06:36.206751 24098 raft_consensus.cc:2454] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 0.
I20260928 21:06:36.207213 24169 leader_election.cc:302] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e58acd871dd45dca8014300c586e1a8, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:36.207492 24364 raft_consensus.cc:2781] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:36.207561 24364 raft_consensus.cc:482] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:36.207629 24364 raft_consensus.cc:3037] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:36.209035 24364 raft_consensus.cc:504] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.209570 24364 leader_election.cc:288] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371)
I20260928 21:06:36.210245 23958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89434938dadc4762b7131a9aa2852f46" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8"
I20260928 21:06:36.210244 24098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89434938dadc4762b7131a9aa2852f46" candidate_uuid: "c6b4acb479f3423d902f2587e063c4b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9170b6db74e85ad9442d91dcc3d08"
I20260928 21:06:36.210424 23958 raft_consensus.cc:3037] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:36.210460 24098 raft_consensus.cc:3037] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:36.211781 23958 raft_consensus.cc:2454] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 1.
I20260928 21:06:36.211788 24098 raft_consensus.cc:2454] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b4acb479f3423d902f2587e063c4b7 in term 1.
I20260928 21:06:36.212319 24169 leader_election.cc:302] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e58acd871dd45dca8014300c586e1a8, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:36.212600 24364 raft_consensus.cc:2781] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:36.212740 24364 raft_consensus.cc:686] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 1 LEADER]: Becoming Leader. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Running, Role: LEADER
I20260928 21:06:36.212929 24364 consensus_queue.cc:237] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.215440 23799 catalog_manager.cc:5495] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "c6b4acb479f3423d902f2587e063c4b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } health_report { overall_health: HEALTHY } } }
I20260928 21:06:36.242765 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:36.238022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.21.140.67:46869)
0928 21:06:36.238430 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:36.238439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:36.238455 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:06:36.238516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:36.238519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:36.238575 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:36.238675 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:36.239360 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.240032 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.241154 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.241482 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.241603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:36.242047 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:36.242158 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:36.242235 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:36.242397 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 21:06:36.242514 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":165,"threads_started":1}
I20260928 21:06:36.252696 24098 raft_consensus.cc:1261] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Refusing update from remote peer c6b4acb479f3423d902f2587e063c4b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:36.252918 23958 raft_consensus.cc:1261] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Refusing update from remote peer c6b4acb479f3423d902f2587e063c4b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:36.253546 24364 consensus_queue.cc:1035] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:36.253921 24382 consensus_queue.cc:1035] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:36.331398 23800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:36.331645 23800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:36.334815 23800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Sending DeleteTablet for 3 replicas of tablet 89434938dadc4762b7131a9aa2852f46
I20260928 21:06:36.336374 24218 tablet_service.cc:1514] Processing DeleteTablet for tablet 89434938dadc4762b7131a9aa2852f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:54168
I20260928 21:06:36.336354 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet 89434938dadc4762b7131a9aa2852f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:48644
I20260928 21:06:36.336676 24388 tablet_replica.cc:330] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: stopping tablet replica
I20260928 21:06:36.336679 24389 tablet_replica.cc:330] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: stopping tablet replica
I20260928 21:06:36.336860 24388 raft_consensus.cc:2229] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:36.336876 24389 raft_consensus.cc:2229] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:36.336959 24071 tablet_service.cc:1514] Processing DeleteTablet for tablet 89434938dadc4762b7131a9aa2852f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:44658
I20260928 21:06:36.337064 24389 raft_consensus.cc:2258] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:36.337131 24388 raft_consensus.cc:2258] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:36.337265 24387 tablet_replica.cc:330] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: stopping tablet replica
I20260928 21:06:36.337441 24387 raft_consensus.cc:2229] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:36.337620 24387 raft_consensus.cc:2258] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:36.337814 23800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:06:36.338300 24388 ts_tablet_manager.cc:1905] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:36.338510 24389 ts_tablet_manager.cc:1905] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:06:36.338562 23800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:36.338621 24387 ts_tablet_manager.cc:1905] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:36.342090 24388 ts_tablet_manager.cc:1918] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:36.342075 24389 ts_tablet_manager.cc:1918] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:36.342226 24389 log.cc:1196] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/89434938dadc4762b7131a9aa2852f46
I20260928 21:06:36.342499 24388 log.cc:1196] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/89434938dadc4762b7131a9aa2852f46
I20260928 21:06:36.342571 24389 ts_tablet_manager.cc:1939] T 89434938dadc4762b7131a9aa2852f46 P 6e58acd871dd45dca8014300c586e1a8: Deleting consensus metadata
I20260928 21:06:36.342631 24387 ts_tablet_manager.cc:1918] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:06:36.342731 24387 log.cc:1196] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/89434938dadc4762b7131a9aa2852f46
I20260928 21:06:36.342815 24388 ts_tablet_manager.cc:1939] T 89434938dadc4762b7131a9aa2852f46 P c6b4acb479f3423d902f2587e063c4b7: Deleting consensus metadata
I20260928 21:06:36.343014 24387 ts_tablet_manager.cc:1939] T 89434938dadc4762b7131a9aa2852f46 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting consensus metadata
I20260928 21:06:36.344493 23786 catalog_manager.cc:4841] TS c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869): tablet 89434938dadc4762b7131a9aa2852f46 (table test-table [id=0c2f53ddf66448bcad902e6a82c8ffa5]) successfully deleted
I20260928 21:06:36.344815 23785 catalog_manager.cc:4841] TS 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367): tablet 89434938dadc4762b7131a9aa2852f46 (table test-table [id=0c2f53ddf66448bcad902e6a82c8ffa5]) successfully deleted
I20260928 21:06:36.344815 23787 catalog_manager.cc:4841] TS 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371): tablet 89434938dadc4762b7131a9aa2852f46 (table test-table [id=0c2f53ddf66448bcad902e6a82c8ffa5]) successfully deleted
I20260928 21:06:36.354254 24218 tablet_service.cc:1467] Processing CreateTablet for tablet c6d83c04e98f4a14bf467b657bd88ef2 (DEFAULT_TABLE table=test-table [id=2cd7e11427ab4884979e5969f02978f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:36.354254 23938 tablet_service.cc:1467] Processing CreateTablet for tablet c6d83c04e98f4a14bf467b657bd88ef2 (DEFAULT_TABLE table=test-table [id=2cd7e11427ab4884979e5969f02978f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:36.354297 24071 tablet_service.cc:1467] Processing CreateTablet for tablet c6d83c04e98f4a14bf467b657bd88ef2 (DEFAULT_TABLE table=test-table [id=2cd7e11427ab4884979e5969f02978f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:36.354990 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d83c04e98f4a14bf467b657bd88ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:36.354991 24071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d83c04e98f4a14bf467b657bd88ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:36.354990 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d83c04e98f4a14bf467b657bd88ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:36.359714 24399 tablet_bootstrap.cc:492] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8: Bootstrap starting.
I20260928 21:06:36.359714 24398 tablet_bootstrap.cc:492] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Bootstrap starting.
I20260928 21:06:36.361408 24399 tablet_bootstrap.cc:654] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:36.361408 24398 tablet_bootstrap.cc:654] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:36.363268 24400 tablet_bootstrap.cc:492] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Bootstrap starting.
I20260928 21:06:36.363724 24398 tablet_bootstrap.cc:492] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: No bootstrap required, opened a new log
I20260928 21:06:36.363914 24398 ts_tablet_manager.cc:1397] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 21:06:36.364338 24399 tablet_bootstrap.cc:492] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8: No bootstrap required, opened a new log
I20260928 21:06:36.364686 24399 ts_tablet_manager.cc:1397] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:06:36.364686 24398 raft_consensus.cc:348] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.364925 24398 raft_consensus.cc:374] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:36.364959 24398 raft_consensus.cc:729] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b4acb479f3423d902f2587e063c4b7, State: Initialized, Role: FOLLOWER
I20260928 21:06:36.365139 24398 consensus_queue.cc:260] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.365249 24400 tablet_bootstrap.cc:654] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:36.365553 24398 ts_tablet_manager.cc:1428] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 21:06:36.365703 24399 raft_consensus.cc:348] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.365868 24399 raft_consensus.cc:374] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:36.365904 24399 raft_consensus.cc:729] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e58acd871dd45dca8014300c586e1a8, State: Initialized, Role: FOLLOWER
I20260928 21:06:36.366226 24399 consensus_queue.cc:260] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.366758 24399 ts_tablet_manager.cc:1428] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:36.367568 24400 tablet_bootstrap.cc:492] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: No bootstrap required, opened a new log
I20260928 21:06:36.367787 24400 ts_tablet_manager.cc:1397] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 21:06:36.368659 24400 raft_consensus.cc:348] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.368875 24400 raft_consensus.cc:374] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:36.368937 24400 raft_consensus.cc:729] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Initialized, Role: FOLLOWER
I20260928 21:06:36.369135 24400 consensus_queue.cc:260] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.369573 24400 ts_tablet_manager.cc:1428] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:06:36.533365 24373 raft_consensus.cc:482] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:36.533597 24373 raft_consensus.cc:504] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.534227 24373 leader_election.cc:288] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869)
I20260928 21:06:36.535005 24238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d83c04e98f4a14bf467b657bd88ef2" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b4acb479f3423d902f2587e063c4b7" is_pre_election: true
I20260928 21:06:36.535116 23957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d83c04e98f4a14bf467b657bd88ef2" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8" is_pre_election: true
I20260928 21:06:36.535230 24238 raft_consensus.cc:2454] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 0.
I20260928 21:06:36.535344 23957 raft_consensus.cc:2454] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 0.
I20260928 21:06:36.535670 24030 leader_election.cc:302] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, 6e58acd871dd45dca8014300c586e1a8, c6b4acb479f3423d902f2587e063c4b7; no voters:
I20260928 21:06:36.535933 24373 raft_consensus.cc:2781] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:36.536017 24373 raft_consensus.cc:482] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:36.536077 24373 raft_consensus.cc:3037] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:36.537755 24373 raft_consensus.cc:504] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.538372 24373 leader_election.cc:288] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 election: Requested vote from peers 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367), c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869)
I20260928 21:06:36.539018 23957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d83c04e98f4a14bf467b657bd88ef2" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58acd871dd45dca8014300c586e1a8"
I20260928 21:06:36.539022 24238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d83c04e98f4a14bf467b657bd88ef2" candidate_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b4acb479f3423d902f2587e063c4b7"
I20260928 21:06:36.539208 24238 raft_consensus.cc:3037] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:36.539216 23957 raft_consensus.cc:3037] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:36.540750 23957 raft_consensus.cc:2454] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 1.
I20260928 21:06:36.541031 24238 raft_consensus.cc:2454] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff9170b6db74e85ad9442d91dcc3d08 in term 1.
I20260928 21:06:36.541539 24029 leader_election.cc:302] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9170b6db74e85ad9442d91dcc3d08, 6e58acd871dd45dca8014300c586e1a8; no voters:
I20260928 21:06:36.541803 24373 raft_consensus.cc:2781] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:36.542064 24373 raft_consensus.cc:686] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 LEADER]: Becoming Leader. State: Replica: 4ff9170b6db74e85ad9442d91dcc3d08, State: Running, Role: LEADER
I20260928 21:06:36.542255 24373 consensus_queue.cc:237] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } }
I20260928 21:06:36.545161 23798 catalog_manager.cc:5495] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66). New cstate: current_term: 1 leader_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ff9170b6db74e85ad9442d91dcc3d08" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 } health_report { overall_health: UNKNOWN } } }
I20260928 21:06:36.576126 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:36.571277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.21.140.66:46371)
0928 21:06:36.571687 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:36.571695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:36.571710 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:06:36.571767 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:36.571770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:36.571824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:36.571924 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:36.572369 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.573030 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.574286 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.574581 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.574716 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:36.575440 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:36.575564 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:36.575649 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:36.575841 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:06:36.575941 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260928 21:06:36.589891 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:36.585342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:55913 (local address 127.21.140.66:46371)
0928 21:06:36.585507 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:36.585512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:36.585735 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:06:36.585829 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:36.585831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:36.585886 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:36.586083 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:36.586789 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.587380 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.588451 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.588719 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.588890 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:36.589166 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:36.589263 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:36.589352 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:36.589542 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 21:06:36.589637 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:06:36.590032 24418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:36.585184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46371 (local address 127.21.140.66:55913)
0928 21:06:36.585590 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:36.585605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:36.585748 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:36.586104 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:36.586107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:36.586349 (+ 242us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:06:36.586539 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:36.586550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.587556 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.587563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:36.588254 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:36.588310 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:36.588927 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:36.588931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:36.589008 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:36.589043 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:06:36.589403 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:06:36.589411 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:06:36.589483 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:06:36.589585 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":196,"threads_started":1}
I20260928 21:06:36.599289 23957 raft_consensus.cc:1261] T c6d83c04e98f4a14bf467b657bd88ef2 P 6e58acd871dd45dca8014300c586e1a8 [term 1 FOLLOWER]: Refusing update from remote peer 4ff9170b6db74e85ad9442d91dcc3d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:36.599380 24238 raft_consensus.cc:1261] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Refusing update from remote peer 4ff9170b6db74e85ad9442d91dcc3d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:36.600113 24419 consensus_queue.cc:1035] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e58acd871dd45dca8014300c586e1a8" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:36.600349 24346 consensus_queue.cc:1035] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b4acb479f3423d902f2587e063c4b7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:06:36.619551 24366 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:06:38.190637 23800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:38.190903 23800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:38.194718 23800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bcad6950a3b42e6aecbe1cbfe1103a9: Sending DeleteTablet for 3 replicas of tablet c6d83c04e98f4a14bf467b657bd88ef2
I20260928 21:06:38.195664 24071 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d83c04e98f4a14bf467b657bd88ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:38 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:44658
I20260928 21:06:38.195695 23938 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d83c04e98f4a14bf467b657bd88ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:38 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:48644
I20260928 21:06:38.196180 24218 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d83c04e98f4a14bf467b657bd88ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:38 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:54168
I20260928 21:06:38.196218 24443 tablet_replica.cc:330] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: stopping tablet replica
I20260928 21:06:38.196448 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 23873
I20260928 21:06:38.196535 24443 raft_consensus.cc:2229] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:38.197919 24443 raft_consensus.cc:2258] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:38.199019 24443 ts_tablet_manager.cc:1905] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:38.199043 24444 tablet_replica.cc:330] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: stopping tablet replica
I20260928 21:06:38.199183 24444 raft_consensus.cc:2229] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:38.199337 24444 raft_consensus.cc:2258] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:38.200038 24444 ts_tablet_manager.cc:1905] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:38.202200 24443 ts_tablet_manager.cc:1918] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:06:38.202330 24443 log.cc:1196] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/c6d83c04e98f4a14bf467b657bd88ef2
I20260928 21:06:38.202683 24443 ts_tablet_manager.cc:1939] T c6d83c04e98f4a14bf467b657bd88ef2 P 4ff9170b6db74e85ad9442d91dcc3d08: Deleting consensus metadata
I20260928 21:06:38.203439 24444 ts_tablet_manager.cc:1918] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:06:38.203549 24444 log.cc:1196] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/c6d83c04e98f4a14bf467b657bd88ef2
I20260928 21:06:38.203862 24444 ts_tablet_manager.cc:1939] T c6d83c04e98f4a14bf467b657bd88ef2 P c6b4acb479f3423d902f2587e063c4b7: Deleting consensus metadata
I20260928 21:06:38.203812 23787 catalog_manager.cc:4841] TS 4ff9170b6db74e85ad9442d91dcc3d08 (127.21.140.66:46371): tablet c6d83c04e98f4a14bf467b657bd88ef2 (table test-table [id=2cd7e11427ab4884979e5969f02978f2]) successfully deleted
I20260928 21:06:38.205140 23786 catalog_manager.cc:4841] TS c6b4acb479f3423d902f2587e063c4b7 (127.21.140.67:46869): tablet c6d83c04e98f4a14bf467b657bd88ef2 (table test-table [id=2cd7e11427ab4884979e5969f02978f2]) successfully deleted
I20260928 21:06:38.208828 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24013
W20260928 21:06:38.208849 23785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:39367 (error 108)
I20260928 21:06:38.213054 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:38.212182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:39367 (local address 127.0.0.1:36856)
0928 21:06:38.212647 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:38.212846 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":153,"threads_started":1}
W20260928 21:06:38.213493 23785 catalog_manager.cc:4584] TS 6e58acd871dd45dca8014300c586e1a8 (127.21.140.65:39367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6d83c04e98f4a14bf467b657bd88ef2: Network error: Client connection negotiation failed: client connection to 127.21.140.65:39367: connect: Connection refused (error 111)
I20260928 21:06:38.219363 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24153
I20260928 21:06:38.228659 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 23769
2026-09-28T21:06:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6802 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:38 dist-test-slave-d5wc krb5kdc[24449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:38 dist-test-slave-d5wc krb5kdc[24449](info): set up 2 sockets
Sep 28 21:06:38 dist-test-slave-d5wc krb5kdc[24449](info): commencing operation
krb5kdc: starting...
W20260928 21:06:39.291419 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.008s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:39 dist-test-slave-d5wc krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:39.422646 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34959
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:39.539492 24462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:39.539827 24462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:39.539915 24462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:39.544314 24462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:39.544416 24462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:39.544456 24462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:39.544515 24462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:39.544570 24462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:39.550196 24462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:34959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:39.551352 24462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:39.552402 24462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:06:39.561527 24462 server_base.cc:1034] running on GCE node
W20260928 21:06:39.561476 24470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:39.561476 24467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:39.561510 24468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:39.562412 24462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:39.563508 24462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:39.564674 24462 hybrid_clock.cc:648] HybridClock initialized: now 1790629599564644 us; error 44 us; skew 500 ppm
Sep 28 21:06:39 dist-test-slave-d5wc krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629599, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:39.567553 24462 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:39.568475 24462 webserver.cc:460] Webserver started at http://127.21.140.126:37465/ using document root <none> and password file <none>
I20260928 21:06:39.569103 24462 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:39.569168 24462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:39.569391 24462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:39.571488 24462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "9c74f2aa4b514c4c974e8e67dbe38d06"
format_stamp: "Formatted at 2026-09-28 21:06:39 on dist-test-slave-d5wc"
server_key: "8c94e6ce0959e7204e5a398f1b7743ff"
server_key_iv: "080fe40a0cde31034b2deb0cd990e70d"
server_key_version: "encryptionkey@0"
I20260928 21:06:39.572341 24462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "9c74f2aa4b514c4c974e8e67dbe38d06"
format_stamp: "Formatted at 2026-09-28 21:06:39 on dist-test-slave-d5wc"
server_key: "8c94e6ce0959e7204e5a398f1b7743ff"
server_key_iv: "080fe40a0cde31034b2deb0cd990e70d"
server_key_version: "encryptionkey@0"
I20260928 21:06:39.576637 24462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:06:39.579329 24477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:39.580626 24462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:39.580734 24462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "9c74f2aa4b514c4c974e8e67dbe38d06"
format_stamp: "Formatted at 2026-09-28 21:06:39 on dist-test-slave-d5wc"
server_key: "8c94e6ce0959e7204e5a398f1b7743ff"
server_key_iv: "080fe40a0cde31034b2deb0cd990e70d"
server_key_version: "encryptionkey@0"
I20260928 21:06:39.580888 24462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:39.591195 24462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:39.594590 24462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:39.594799 24462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:39.602326 24462 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:34959
I20260928 21:06:39.602352 24529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:34959 every 8 connection(s)
I20260928 21:06:39.603298 24462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:39.606169 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:39.609651 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24462
I20260928 21:06:39.609742 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:39.610082 22065 external_mini_cluster.cc:1428] Setting key a6becce42373cd0a647013a5315d69d5
I20260928 21:06:39.612246 24530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06: Bootstrap starting.
I20260928 21:06:39.614899 24530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:39.615749 24530 log.cc:824] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:39.618003 24530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06: No bootstrap required, opened a new log
Sep 28 21:06:39 dist-test-slave-d5wc krb5kdc[24449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:39.622401 24530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34959 } }
I20260928 21:06:39.622743 24530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:39.622800 24530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c74f2aa4b514c4c974e8e67dbe38d06, State: Initialized, Role: FOLLOWER
I20260928 21:06:39.623317 24530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34959 } }
I20260928 21:06:39.623486 24530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:39.623569 24530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:39.623667 24530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:39.624796 24530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34959 } }
I20260928 21:06:39.625202 24530 leader_election.cc:302] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c74f2aa4b514c4c974e8e67dbe38d06; no voters:
I20260928 21:06:39.625571 24530 leader_election.cc:288] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:39.625681 24535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:39.625900 24535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [term 1 LEADER]: Becoming Leader. State: Replica: 9c74f2aa4b514c4c974e8e67dbe38d06, State: Running, Role: LEADER
I20260928 21:06:39.626762 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:39.611645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.21.140.126:34959)
0928 21:06:39.612089 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:39.612108 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:39.612162 (+ 54us) server_negotiation.cc:407] Connection header received
0928 21:06:39.612873 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:39.612894 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:39.613251 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:39.613544 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:39.614505 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:39.615604 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:39.616682 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:39.617186 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:39.617308 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:39.620153 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:39.620160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:39.620177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:39.620210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:39.623002 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:39.623541 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:39.623547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:39.623566 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:39.623693 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:39.624068 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:39.624072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:39.624074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:39.624498 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:39.624679 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:39.625076 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 21:06:39.625428 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":161,"threads_started":1}
I20260928 21:06:39.626895 24530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:39.626959 24535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34959 } }
I20260928 21:06:39.629575 24537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c74f2aa4b514c4c974e8e67dbe38d06. Latest consensus state: current_term: 1 leader_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34959 } } }
I20260928 21:06:39.629662 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c74f2aa4b514c4c974e8e67dbe38d06" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34959 } } }
I20260928 21:06:39.629729 24537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:39.629729 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:39.630191 24550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:39.632889 24550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:39.639570 24550 catalog_manager.cc:1347] Generated new cluster ID: c828d1856f8949b998a017b7fb302321
I20260928 21:06:39.639775 24550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:39.656983 24550 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:39.658402 24550 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:39.668325 24550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c74f2aa4b514c4c974e8e67dbe38d06: Generated new TSK 0
I20260928 21:06:39.669200 24550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:39.702826 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34959
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:39.816504 24556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:39.816825 24556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:39.816921 24556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:39.821135 24556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:39.821244 24556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:39.821350 24556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:39.828351 24556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:39.829605 24556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:39.830719 24556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:39.840581 24561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:39.840597 24562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:39.840610 24564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:39.840757 24556 server_base.cc:1034] running on GCE node
I20260928 21:06:39.841257 24556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:39.841919 24556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:39.843163 24556 hybrid_clock.cc:648] HybridClock initialized: now 1790629599843123 us; error 51 us; skew 500 ppm
Sep 28 21:06:39 dist-test-slave-d5wc krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629599, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:39.846094 24556 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:39.847012 24556 webserver.cc:460] Webserver started at http://127.21.140.65:41499/ using document root <none> and password file <none>
I20260928 21:06:39.847587 24556 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:39.847689 24556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:39.847922 24556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:39.850073 24556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "e8bf025c6a024a708fbe93d067026619"
format_stamp: "Formatted at 2026-09-28 21:06:39 on dist-test-slave-d5wc"
server_key: "2a6c01030bbf1a0a8460b6d670c2570f"
server_key_iv: "9e0020921fe5ee02160b558cc4be5f01"
server_key_version: "encryptionkey@0"
I20260928 21:06:39.850642 24556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "e8bf025c6a024a708fbe93d067026619"
format_stamp: "Formatted at 2026-09-28 21:06:39 on dist-test-slave-d5wc"
server_key: "2a6c01030bbf1a0a8460b6d670c2570f"
server_key_iv: "9e0020921fe5ee02160b558cc4be5f01"
server_key_version: "encryptionkey@0"
I20260928 21:06:39.854488 24556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 21:06:39.857234 24571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:39.858723 24556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:39.858886 24556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "e8bf025c6a024a708fbe93d067026619"
format_stamp: "Formatted at 2026-09-28 21:06:39 on dist-test-slave-d5wc"
server_key: "2a6c01030bbf1a0a8460b6d670c2570f"
server_key_iv: "9e0020921fe5ee02160b558cc4be5f01"
server_key_version: "encryptionkey@0"
I20260928 21:06:39.859025 24556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:39.884766 24556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:39.888942 24556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:39.889204 24556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:39.889899 24556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:39.890878 24556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:39.890961 24556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:39.891036 24556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:39.891090 24556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:39.902002 24556 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:41865
I20260928 21:06:39.902055 24684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:41865 every 8 connection(s)
I20260928 21:06:39.903008 24556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:39.910262 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24556
I20260928 21:06:39.910358 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:39.910666 22065 external_mini_cluster.cc:1428] Setting key 00462b2921953020ae4a9cfc5ae87d25
Sep 28 21:06:39 dist-test-slave-d5wc krb5kdc[24449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629599, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:39.918387 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:39.905117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:52613 (local address 127.21.140.126:34959)
0928 21:06:39.905277 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:39.905281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:39.906125 (+ 844us) server_negotiation.cc:407] Connection header received
0928 21:06:39.907442 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:39.907446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:39.907496 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:39.907601 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:39.908982 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:39.909538 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:39.911062 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:39.911361 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:39.911514 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:39.914432 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:39.914435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:39.914437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:39.914462 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:39.916009 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:39.916677 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:39.916679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:39.916681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:39.916756 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:39.917178 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:39.917180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:39.917181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:39.917353 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:39.917425 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:39.918162 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 21:06:39.918267 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:06:39.919621 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:39.905445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34959 (local address 127.21.140.65:52613)
0928 21:06:39.905904 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:39.905947 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:39.907236 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:39.907823 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:39.907831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:39.908224 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:39.908803 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:39.908819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:39.909661 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:39.909664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:39.910909 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:39.910920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:39.911556 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:39.911560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:39.911716 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:39.912397 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:39.912419 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:39.914238 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:39.916139 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:39.916146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:39.916165 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:39.916559 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:39.916851 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:39.916856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:39.916861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:39.917071 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:39.917462 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:39.917472 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:39.917841 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 21:06:39.918114 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 21:06:39.918488 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:39.920976 24685 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34959
I20260928 21:06:39.921201 24685 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:39.921692 24685 heartbeater.cc:511] Master 127.21.140.126:34959 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:06:39.923317 24494 ts_manager.cc:194] Registered new tserver with Master: e8bf025c6a024a708fbe93d067026619 (127.21.140.65:41865)
I20260928 21:06:39.924660 24494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:52613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:39.943204 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34959
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:40.059152 24690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:40.059437 24690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:40.059484 24690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:40.063813 24690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:40.063882 24690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:40.063943 24690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:40.069384 24690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:40.070529 24690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:40.071574 24690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:40.081264 24698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:40.081390 24690 server_base.cc:1034] running on GCE node
W20260928 21:06:40.081313 24695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:40.081593 24696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:40.082041 24690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:40.082718 24690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:40.083877 24690 hybrid_clock.cc:648] HybridClock initialized: now 1790629600083853 us; error 36 us; skew 500 ppm
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:40.087074 24690 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:40.087972 24690 webserver.cc:460] Webserver started at http://127.21.140.66:46027/ using document root <none> and password file <none>
I20260928 21:06:40.088539 24690 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:40.088600 24690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:40.088830 24690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:40.091095 24690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "fdf080a0490a4da199db9b7d88f62f3c"
format_stamp: "Formatted at 2026-09-28 21:06:40 on dist-test-slave-d5wc"
server_key: "b6b81bdaba534611efc831e6dbac0a65"
server_key_iv: "f35454c202e2b691a9209535eb618986"
server_key_version: "encryptionkey@0"
I20260928 21:06:40.091672 24690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "fdf080a0490a4da199db9b7d88f62f3c"
format_stamp: "Formatted at 2026-09-28 21:06:40 on dist-test-slave-d5wc"
server_key: "b6b81bdaba534611efc831e6dbac0a65"
server_key_iv: "f35454c202e2b691a9209535eb618986"
server_key_version: "encryptionkey@0"
I20260928 21:06:40.095552 24690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:40.098088 24705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:40.099310 24690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:40.099436 24690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "fdf080a0490a4da199db9b7d88f62f3c"
format_stamp: "Formatted at 2026-09-28 21:06:40 on dist-test-slave-d5wc"
server_key: "b6b81bdaba534611efc831e6dbac0a65"
server_key_iv: "f35454c202e2b691a9209535eb618986"
server_key_version: "encryptionkey@0"
I20260928 21:06:40.099551 24690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:40.121063 24690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:40.124048 24690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:40.124267 24690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:40.124876 24690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:40.125862 24690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:40.125909 24690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:40.125974 24690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:40.126049 24690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:40.135707 24690 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:43697
I20260928 21:06:40.135744 24818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:43697 every 8 connection(s)
I20260928 21:06:40.136621 24690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:40.140486 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24690
I20260928 21:06:40.140581 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:40.140864 22065 external_mini_cluster.cc:1428] Setting key 9c9231f090796c3bc5e21bccf186204f
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:40.153190 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:40.138544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:53923 (local address 127.21.140.126:34959)
0928 21:06:40.138700 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:40.138703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:40.139565 (+ 862us) server_negotiation.cc:407] Connection header received
0928 21:06:40.141022 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:40.141026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:40.141088 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:40.141187 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:40.143120 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.143968 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.145284 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.145661 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.145753 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:40.148923 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:40.148926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:40.148928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:40.148953 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:40.150555 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:40.151099 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:40.151102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:40.151104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:40.151189 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:40.151716 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:40.151720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:40.151722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:40.151941 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:40.152276 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:40.152862 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 21:06:40.153009 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:06:40.154348 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:40.138957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34959 (local address 127.21.140.66:53923)
0928 21:06:40.139415 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:40.139457 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:40.140760 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:40.141700 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:40.141715 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:40.142292 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:40.142955 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:40.142972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.144130 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.144133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:40.145143 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:40.145154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.145850 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.145854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:40.146135 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:40.146853 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:40.146870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:40.148724 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:40.150689 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:40.150696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:40.150715 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:40.150983 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:40.151291 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:40.151295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:40.151297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:40.151490 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:40.152260 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:40.152268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:40.152578 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 21:06:40.152790 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 21:06:40.153158 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20260928 21:06:40.155877 24819 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34959
I20260928 21:06:40.156162 24819 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:40.156700 24819 heartbeater.cc:511] Master 127.21.140.126:34959 requested a full tablet report, sending...
I20260928 21:06:40.157823 24494 ts_manager.cc:194] Registered new tserver with Master: fdf080a0490a4da199db9b7d88f62f3c (127.21.140.66:43697)
I20260928 21:06:40.158741 24494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:53923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:40.174496 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34959
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:40.292961 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:40.293278 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:40.293355 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:40.297714 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:40.297821 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:40.297919 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:40.303570 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:40.304702 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:40.305748 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:40.315611 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:40.315842 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:40.316222 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:40.316278 24824 server_base.cc:1034] running on GCE node
I20260928 21:06:40.316752 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:40.317788 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:40.319005 24824 hybrid_clock.cc:648] HybridClock initialized: now 1790629600318964 us; error 55 us; skew 500 ppm
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:40.323347 24824 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:40.324760 24824 webserver.cc:460] Webserver started at http://127.21.140.67:42075/ using document root <none> and password file <none>
I20260928 21:06:40.325555 24824 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:40.325691 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:40.326153 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:40.329743 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "667d273b1f5c4b2c921ce8fb4ada6453"
format_stamp: "Formatted at 2026-09-28 21:06:40 on dist-test-slave-d5wc"
server_key: "b5964cac7db2909014fb7704925e5801"
server_key_iv: "fbedca047f00815dfc913ce4ffad12fc"
server_key_version: "encryptionkey@0"
I20260928 21:06:40.330652 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "667d273b1f5c4b2c921ce8fb4ada6453"
format_stamp: "Formatted at 2026-09-28 21:06:40 on dist-test-slave-d5wc"
server_key: "b5964cac7db2909014fb7704925e5801"
server_key_iv: "fbedca047f00815dfc913ce4ffad12fc"
server_key_version: "encryptionkey@0"
I20260928 21:06:40.336404 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 21:06:40.339885 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:40.341503 24824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:40.341651 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "667d273b1f5c4b2c921ce8fb4ada6453"
format_stamp: "Formatted at 2026-09-28 21:06:40 on dist-test-slave-d5wc"
server_key: "b5964cac7db2909014fb7704925e5801"
server_key_iv: "fbedca047f00815dfc913ce4ffad12fc"
server_key_version: "encryptionkey@0"
I20260928 21:06:40.341796 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:40.362270 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:40.366258 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:40.366415 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:40.366957 24824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:40.367759 24824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:40.367807 24824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:40.367853 24824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:40.367869 24824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:40.377071 24824 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:43935
I20260928 21:06:40.377095 24952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:43935 every 8 connection(s)
I20260928 21:06:40.378070 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:40.382140 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24824
I20260928 21:06:40.382282 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:40.382622 22065 external_mini_cluster.cc:1428] Setting key 9fbc66865798baba3ed15d2eb874722b
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:40.394380 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:40.379923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:45673 (local address 127.21.140.126:34959)
0928 21:06:40.380116 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:40.380120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:40.381079 (+ 959us) server_negotiation.cc:407] Connection header received
0928 21:06:40.382912 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:40.382915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:40.383105 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:40.383206 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:40.384968 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.385662 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.387117 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.387343 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.387520 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:40.390281 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:40.390285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:40.390287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:40.390312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:40.391783 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:40.392337 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:40.392340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:40.392489 (+ 149us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:40.392591 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:40.393029 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:40.393032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:40.393033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:40.393256 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:40.393341 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:40.394112 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 21:06:40.394243 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:06:40.395320 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:40.380322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34959 (local address 127.21.140.67:45673)
0928 21:06:40.380849 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:40.380899 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:40.382639 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:40.383492 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:40.383501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:40.384135 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:40.384824 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:40.384839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.385823 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.385826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:40.386957 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:40.386976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:40.387547 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:40.387550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:40.387725 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:40.388361 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:40.388377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:40.390081 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:40.391908 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:40.391913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:40.391927 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:40.392199 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:40.392705 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:40.392708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:40.392709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:40.392856 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:40.393398 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:40.393404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:40.393678 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 21:06:40.393879 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 21:06:40.394267 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260928 21:06:40.396750 24953 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34959
I20260928 21:06:40.397027 24953 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:40.397541 24953 heartbeater.cc:511] Master 127.21.140.126:34959 requested a full tablet report, sending...
I20260928 21:06:40.398672 24494 ts_manager.cc:194] Registered new tserver with Master: 667d273b1f5c4b2c921ce8fb4ada6453 (127.21.140.67:43935)
I20260928 21:06:40.399286 24494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:45673
I20260928 21:06:40.409284 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:06:40.424100 24533 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:40.422496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.21.140.126:34959)
0928 21:06:40.422662 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:40.422667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:40.422837 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:06:40.422953 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:40.422956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:40.423002 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:40.423084 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:40.423902 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58936: BlockingRecv error: recv got EOF from 127.0.0.1:58936 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 21:06:40.425056 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24556
I20260928 21:06:40.431285 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24690
I20260928 21:06:40.437117 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24824
I20260928 21:06:40.442749 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24462
2026-09-28T21:06:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2209 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24965](info): set up 2 sockets
Sep 28 21:06:40 dist-test-slave-d5wc krb5kdc[24965](info): commencing operation
krb5kdc: starting...
W20260928 21:06:41.495496 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:41 dist-test-slave-d5wc krb5kdc[24965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:41.621551 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44953
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:41.730831 24978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:41.731102 24978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:41.731144 24978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:41.735168 24978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:41.735226 24978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:41.735246 24978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:41.735263 24978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:41.735278 24978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:41.740512 24978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:41.741533 24978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:41.742486 24978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:41.750805 24984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:41.750885 24983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:41.750937 24978 server_base.cc:1034] running on GCE node
W20260928 21:06:41.751159 24986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:41.751722 24978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:41.752795 24978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:41.754042 24978 hybrid_clock.cc:648] HybridClock initialized: now 1790629601754009 us; error 42 us; skew 500 ppm
Sep 28 21:06:41 dist-test-slave-d5wc krb5kdc[24965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629601, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:41.763191 24978 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:41.764144 24978 webserver.cc:460] Webserver started at http://127.21.140.126:37155/ using document root <none> and password file <none>
I20260928 21:06:41.764711 24978 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:41.764772 24978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:41.764947 24978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:41.767114 24978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "83868b021d1241e2844b46d20e2952f1"
format_stamp: "Formatted at 2026-09-28 21:06:41 on dist-test-slave-d5wc"
server_key: "5d4fb9a88f5628a12ba4090fbbf150cb"
server_key_iv: "66423a65048d3822937b55afc4efe1d4"
server_key_version: "encryptionkey@0"
I20260928 21:06:41.767669 24978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "83868b021d1241e2844b46d20e2952f1"
format_stamp: "Formatted at 2026-09-28 21:06:41 on dist-test-slave-d5wc"
server_key: "5d4fb9a88f5628a12ba4090fbbf150cb"
server_key_iv: "66423a65048d3822937b55afc4efe1d4"
server_key_version: "encryptionkey@0"
I20260928 21:06:41.771301 24978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:06:41.773600 24993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:41.774878 24978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:41.775003 24978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "83868b021d1241e2844b46d20e2952f1"
format_stamp: "Formatted at 2026-09-28 21:06:41 on dist-test-slave-d5wc"
server_key: "5d4fb9a88f5628a12ba4090fbbf150cb"
server_key_iv: "66423a65048d3822937b55afc4efe1d4"
server_key_version: "encryptionkey@0"
I20260928 21:06:41.775106 24978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:41.810734 24978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:41.813625 24978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:41.813834 24978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:41.820920 24978 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44953
I20260928 21:06:41.821264 25045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44953 every 8 connection(s)
I20260928 21:06:41.821950 24978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:41.824769 25046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:41.828228 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 24978
I20260928 21:06:41.828321 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:41.828580 22065 external_mini_cluster.cc:1428] Setting key 77659382a57c028b018e232591db7ae1
I20260928 21:06:41.830982 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1: Bootstrap starting.
I20260928 21:06:41.833876 25046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:41.834775 25046 log.cc:824] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:41.836941 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1: No bootstrap required, opened a new log
Sep 28 21:06:41 dist-test-slave-d5wc krb5kdc[24965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:41.840657 25046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83868b021d1241e2844b46d20e2952f1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44953 } }
I20260928 21:06:41.840973 25046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:41.841060 25046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83868b021d1241e2844b46d20e2952f1, State: Initialized, Role: FOLLOWER
I20260928 21:06:41.841532 25046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83868b021d1241e2844b46d20e2952f1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44953 } }
I20260928 21:06:41.841656 25046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:41.841737 25046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:41.841836 25046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:41.842733 25046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83868b021d1241e2844b46d20e2952f1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44953 } }
I20260928 21:06:41.843075 25046 leader_election.cc:302] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83868b021d1241e2844b46d20e2952f1; no voters:
I20260928 21:06:41.843418 25046 leader_election.cc:288] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:41.843565 25051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:41.843787 25051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [term 1 LEADER]: Becoming Leader. State: Replica: 83868b021d1241e2844b46d20e2952f1, State: Running, Role: LEADER
I20260928 21:06:41.844098 25051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83868b021d1241e2844b46d20e2952f1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44953 } }
I20260928 21:06:41.844743 25046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:41.844959 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:41.830040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.21.140.126:44953)
0928 21:06:41.830520 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:41.830540 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:41.830582 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:06:41.831460 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:41.831487 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:41.831979 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:41.832371 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:41.833207 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:41.834775 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:41.835900 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:41.836565 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:41.836733 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:41.839401 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:41.839405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:41.839418 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:41.839443 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:41.841691 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:41.842241 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:41.842247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:41.842252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:41.842370 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:41.842668 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:41.842671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:41.842673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:41.843133 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:41.843331 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:41.843671 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:06:41.844116 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
I20260928 21:06:41.846289 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83868b021d1241e2844b46d20e2952f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83868b021d1241e2844b46d20e2952f1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44953 } } }
I20260928 21:06:41.846432 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:41.846640 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83868b021d1241e2844b46d20e2952f1. Latest consensus state: current_term: 1 leader_uuid: "83868b021d1241e2844b46d20e2952f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83868b021d1241e2844b46d20e2952f1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44953 } } }
I20260928 21:06:41.846727 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:41.848234 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 21:06:41.849649 25066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:06:41.849715 25066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:06:41.849792 25064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:41.850741 25064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:41.856648 25064 catalog_manager.cc:1347] Generated new cluster ID: 72f78f63c7434c6fb6b48a36aa71cff0
I20260928 21:06:41.856721 25064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:41.875514 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:41.869762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.21.140.126:44953)
0928 21:06:41.869885 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:41.869890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:41.870167 (+ 277us) server_negotiation.cc:407] Connection header received
0928 21:06:41.870312 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:41.870316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:41.870369 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:41.870451 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:41.871025 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:41.871850 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:41.872978 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:41.873355 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:41.873460 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:41.874396 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:41.874400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:41.874403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:41.875041 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:41.875077 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:41.875097 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:41.875192 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:41.875298 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:06:41.875361 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:06:41.891402 25064 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:41.892427 25064 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:41.906495 25064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83868b021d1241e2844b46d20e2952f1: Generated new TSK 0
I20260928 21:06:41.907451 25064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:06:41.920316 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 24978
2026-09-28T21:06:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1475 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:41 dist-test-slave-d5wc krb5kdc[25079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:41 dist-test-slave-d5wc krb5kdc[25079](info): set up 2 sockets
Sep 28 21:06:41 dist-test-slave-d5wc krb5kdc[25079](info): commencing operation
krb5kdc: starting...
W20260928 21:06:43.047145 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 21:06:43.237543 22065 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.190s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 21:06:43.418735 22065 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.181s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 21:06:43.590559 22065 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.172s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/test-user.keytab.
W20260928 21:06:44.351154 22065 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.760s user 0.001s sys 0.002s
Sep 28 21:06:44 dist-test-slave-d5wc krb5kdc[25079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
W20260928 21:06:45.795750 22065 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.140.126: real 1.414s user 0.002s sys 0.003s
I20260928 21:06:45.796604 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37789
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:45.947614 25092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:45.947927 25092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:45.947973 25092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:45.952447 25092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:45.952533 25092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:45.952554 25092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:45.952574 25092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:45.952590 25092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:45.958317 25092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:45.959589 25092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:45.960762 25092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:45.970315 25098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:45.970551 25092 server_base.cc:1034] running on GCE node
W20260928 21:06:45.971617 25097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:45.971864 25100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:45.972540 25092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:45.973795 25092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:45.974987 25092 hybrid_clock.cc:648] HybridClock initialized: now 1790629605974959 us; error 44 us; skew 500 ppm
Sep 28 21:06:45 dist-test-slave-d5wc krb5kdc[25079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629605, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:45.978091 25092 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:45.979116 25092 webserver.cc:460] Webserver started at http://127.21.140.126:40809/ using document root <none> and password file <none>
I20260928 21:06:45.979735 25092 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:45.979806 25092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:45.980043 25092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:45.982347 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "1463eb9cd7064c3cb60ab97677de7caa"
format_stamp: "Formatted at 2026-09-28 21:06:45 on dist-test-slave-d5wc"
server_key: "1fd7f4476ab07cdf9bd9d0e66b6e8071"
server_key_iv: "fa8e572ce67155e899d2560e6f6713b9"
server_key_version: "encryptionkey@0"
I20260928 21:06:45.982949 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "1463eb9cd7064c3cb60ab97677de7caa"
format_stamp: "Formatted at 2026-09-28 21:06:45 on dist-test-slave-d5wc"
server_key: "1fd7f4476ab07cdf9bd9d0e66b6e8071"
server_key_iv: "fa8e572ce67155e899d2560e6f6713b9"
server_key_version: "encryptionkey@0"
I20260928 21:06:45.987149 25092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:45.990206 25107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:45.991662 25092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:06:45.991775 25092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "1463eb9cd7064c3cb60ab97677de7caa"
format_stamp: "Formatted at 2026-09-28 21:06:45 on dist-test-slave-d5wc"
server_key: "1fd7f4476ab07cdf9bd9d0e66b6e8071"
server_key_iv: "fa8e572ce67155e899d2560e6f6713b9"
server_key_version: "encryptionkey@0"
I20260928 21:06:45.991937 25092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:46.014351 25092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:46.017755 25092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:46.017928 25092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:46.025309 25092 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37789
I20260928 21:06:46.025646 25159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37789 every 8 connection(s)
I20260928 21:06:46.026401 25092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:46.029678 25160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:46.034282 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25092
I20260928 21:06:46.034433 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:46.034716 22065 external_mini_cluster.cc:1428] Setting key 35fdde6d409a56f5b1f3facc4144aa5b
I20260928 21:06:46.036321 25160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa: Bootstrap starting.
I20260928 21:06:46.039336 25160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:46.040319 25160 log.cc:824] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:46.042618 25160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa: No bootstrap required, opened a new log
Sep 28 21:06:46 dist-test-slave-d5wc krb5kdc[25079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:46.047304 25160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1463eb9cd7064c3cb60ab97677de7caa" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37789 } }
I20260928 21:06:46.047606 25160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:46.047662 25160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1463eb9cd7064c3cb60ab97677de7caa, State: Initialized, Role: FOLLOWER
I20260928 21:06:46.048175 25160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1463eb9cd7064c3cb60ab97677de7caa" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37789 } }
I20260928 21:06:46.048301 25160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:46.048436 25160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:46.048554 25160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:46.050993 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:46.036247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.21.140.126:37789)
0928 21:06:46.036717 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:46.036736 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:46.036781 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:06:46.037710 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:46.037735 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:46.038233 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:46.038629 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:46.039369 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.040877 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.041822 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.042526 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.042688 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:46.045575 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:46.045585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:46.045608 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:46.045650 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:46.047879 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:46.048394 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:46.048403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:46.048409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:46.048567 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:46.048859 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:46.048865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:46.048869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:46.049380 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:46.049587 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:46.049829 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 21:06:46.050201 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":168,"threads_started":1}
I20260928 21:06:46.104178 25160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1463eb9cd7064c3cb60ab97677de7caa" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37789 } }
I20260928 21:06:46.104714 25160 leader_election.cc:302] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1463eb9cd7064c3cb60ab97677de7caa; no voters:
I20260928 21:06:46.105122 25160 leader_election.cc:288] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:46.105269 25165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:46.105527 25165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [term 1 LEADER]: Becoming Leader. State: Replica: 1463eb9cd7064c3cb60ab97677de7caa, State: Running, Role: LEADER
I20260928 21:06:46.106003 25165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1463eb9cd7064c3cb60ab97677de7caa" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37789 } }
I20260928 21:06:46.106720 25160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:46.107944 25167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1463eb9cd7064c3cb60ab97677de7caa. Latest consensus state: current_term: 1 leader_uuid: "1463eb9cd7064c3cb60ab97677de7caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1463eb9cd7064c3cb60ab97677de7caa" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37789 } } }
I20260928 21:06:46.108104 25167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:46.108457 25166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1463eb9cd7064c3cb60ab97677de7caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1463eb9cd7064c3cb60ab97677de7caa" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37789 } } }
I20260928 21:06:46.108570 25166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa [sys.catalog]: This master's current role is: LEADER
W20260928 21:06:46.112339 25180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:06:46.112411 25180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:06:46.112493 25174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:46.113350 25174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:46.120150 25174 catalog_manager.cc:1347] Generated new cluster ID: 26f590872b5a467f81cb53fffc2436c3
I20260928 21:06:46.120230 25174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:46.147117 25174 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:46.148154 25174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:46.157823 25174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1463eb9cd7064c3cb60ab97677de7caa: Generated new TSK 0
I20260928 21:06:46.159440 25174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
W20260928 21:06:46.497841 22065 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.140.65: real 0.326s user 0.002s sys 0.002s
I20260928 21:06:46.498459 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37789
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:46.612871 25186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:46.613371 25186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:46.613507 25186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:46.617769 25186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:46.617866 25186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:46.618005 25186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:46.623394 25186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:46.624470 25186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:46.625514 25186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:46.635524 25194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:46.635524 25191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:46.635576 25192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:46.635751 25186 server_base.cc:1034] running on GCE node
I20260928 21:06:46.636281 25186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:46.637056 25186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:46.638283 25186 hybrid_clock.cc:648] HybridClock initialized: now 1790629606638264 us; error 33 us; skew 500 ppm
Sep 28 21:06:46 dist-test-slave-d5wc krb5kdc[25079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:46.641258 25186 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:46.642467 25186 webserver.cc:460] Webserver started at http://127.21.140.65:38815/ using document root <none> and password file <none>
I20260928 21:06:46.643085 25186 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:46.643180 25186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:46.643424 25186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:46.645545 25186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "b7f1de108beb49dfb4844caf0bd17357"
format_stamp: "Formatted at 2026-09-28 21:06:46 on dist-test-slave-d5wc"
server_key: "7e5ee898856f3f846e02d3ace4d5e1d5"
server_key_iv: "9688d16cdc5698747e455bc16147b950"
server_key_version: "encryptionkey@0"
I20260928 21:06:46.646210 25186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "b7f1de108beb49dfb4844caf0bd17357"
format_stamp: "Formatted at 2026-09-28 21:06:46 on dist-test-slave-d5wc"
server_key: "7e5ee898856f3f846e02d3ace4d5e1d5"
server_key_iv: "9688d16cdc5698747e455bc16147b950"
server_key_version: "encryptionkey@0"
I20260928 21:06:46.650214 25186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:46.652861 25201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:46.654444 25186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:46.654589 25186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "b7f1de108beb49dfb4844caf0bd17357"
format_stamp: "Formatted at 2026-09-28 21:06:46 on dist-test-slave-d5wc"
server_key: "7e5ee898856f3f846e02d3ace4d5e1d5"
server_key_iv: "9688d16cdc5698747e455bc16147b950"
server_key_version: "encryptionkey@0"
I20260928 21:06:46.654716 25186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:46.672583 25186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:46.676060 25186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:46.676275 25186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:46.676887 25186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:46.677723 25186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:46.677796 25186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:46.677870 25186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:46.677913 25186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:46.687026 25186 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:36157
I20260928 21:06:46.687076 25314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:36157 every 8 connection(s)
I20260928 21:06:46.687997 25186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:46.695123 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25186
I20260928 21:06:46.695267 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:46.695545 22065 external_mini_cluster.cc:1428] Setting key 5474c2b2af4515ae4428f986ceffcbff
Sep 28 21:06:46 dist-test-slave-d5wc krb5kdc[25079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:46.703725 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:46.689925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:41921 (local address 127.21.140.126:37789)
0928 21:06:46.690199 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:46.690203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:46.691048 (+ 845us) server_negotiation.cc:407] Connection header received
0928 21:06:46.692408 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:46.692412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:46.692475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:46.692599 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:46.693769 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.694391 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.695969 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.696309 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.696421 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:46.699417 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:46.699420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:46.699423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:46.699447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:46.701128 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:46.701857 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:46.701860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:46.701861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:46.701943 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:46.702460 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:46.702462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:46.702463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:46.702660 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:46.702782 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:46.703461 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 21:06:46.703592 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260928 21:06:46.704973 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:46.690388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37789 (local address 127.21.140.65:41921)
0928 21:06:46.690864 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:46.690906 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:46.692189 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:46.692774 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:46.692781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:46.693141 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:46.693605 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:46.693617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.694529 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.694532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:46.695826 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:46.695837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.696473 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.696476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:46.696625 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:46.697267 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:46.697284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:46.699208 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:46.701310 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:46.701321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:46.701343 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:46.701741 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:46.702133 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:46.702138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:46.702144 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:46.702350 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:46.702788 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:46.702797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:46.703170 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 21:06:46.703408 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 21:06:46.703805 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:46.706526 25315 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37789
I20260928 21:06:46.706875 25315 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:46.707504 25315 heartbeater.cc:511] Master 127.21.140.126:37789 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:06:46.709331 25121 ts_manager.cc:194] Registered new tserver with Master: b7f1de108beb49dfb4844caf0bd17357 (127.21.140.65:36157)
I20260928 21:06:46.711063 25121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:41921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:46.726537 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37789
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:46.838986 25321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:46.839244 25321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:46.839287 25321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:46.843513 25321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:46.843580 25321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:46.843638 25321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:46.849016 25321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:46.850129 25321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:46.851166 25321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:46.862116 25327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:46.862116 25329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:46.862121 25326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:46.862666 25321 server_base.cc:1034] running on GCE node
I20260928 21:06:46.863114 25321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:46.863785 25321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:46.865013 25321 hybrid_clock.cc:648] HybridClock initialized: now 1790629606864937 us; error 31 us; skew 500 ppm
Sep 28 21:06:46 dist-test-slave-d5wc krb5kdc[25079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:46.874318 25321 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:46.875240 25321 webserver.cc:460] Webserver started at http://127.21.140.66:34057/ using document root <none> and password file <none>
I20260928 21:06:46.875774 25321 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:46.875833 25321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:46.876011 25321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:46.878103 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "bb9f4340426e485d8979d302ee290a0e"
format_stamp: "Formatted at 2026-09-28 21:06:46 on dist-test-slave-d5wc"
server_key: "ad6a12a26f069b8cdab554c858641ebb"
server_key_iv: "c96ac35aac6495547593072750162c84"
server_key_version: "encryptionkey@0"
I20260928 21:06:46.878645 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "bb9f4340426e485d8979d302ee290a0e"
format_stamp: "Formatted at 2026-09-28 21:06:46 on dist-test-slave-d5wc"
server_key: "ad6a12a26f069b8cdab554c858641ebb"
server_key_iv: "c96ac35aac6495547593072750162c84"
server_key_version: "encryptionkey@0"
I20260928 21:06:46.882452 25321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:06:46.884850 25336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:46.886193 25321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:46.886333 25321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "bb9f4340426e485d8979d302ee290a0e"
format_stamp: "Formatted at 2026-09-28 21:06:46 on dist-test-slave-d5wc"
server_key: "ad6a12a26f069b8cdab554c858641ebb"
server_key_iv: "c96ac35aac6495547593072750162c84"
server_key_version: "encryptionkey@0"
I20260928 21:06:46.886445 25321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:46.901153 25321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:46.904452 25321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:46.904685 25321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:46.905347 25321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:46.906306 25321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:46.906353 25321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:46.906416 25321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:46.906448 25321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:46.915654 25321 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:39633
I20260928 21:06:46.915689 25449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:39633 every 8 connection(s)
I20260928 21:06:46.916539 25321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:46.923322 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25321
I20260928 21:06:46.923476 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:46.923775 22065 external_mini_cluster.cc:1428] Setting key 87403888452cb1a6f09f7ee2724e3491
Sep 28 21:06:46 dist-test-slave-d5wc krb5kdc[25079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:46.931237 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:46.918444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:56231 (local address 127.21.140.126:37789)
0928 21:06:46.918601 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:46.918605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:46.919470 (+ 865us) server_negotiation.cc:407] Connection header received
0928 21:06:46.920855 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:46.920859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:46.920917 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:46.921028 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:46.922274 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.922851 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.924146 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.924451 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.924563 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:46.927398 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:46.927403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:46.927405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:46.927439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:46.929024 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:46.929635 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:46.929638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:46.929639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:46.929716 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:46.930116 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:46.930119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:46.930121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:46.930312 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:46.930423 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:46.930930 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 21:06:46.931063 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:06:46.932053 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:46.918790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37789 (local address 127.21.140.66:56231)
0928 21:06:46.919288 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:46.919331 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:46.920634 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:46.921204 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:46.921211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:46.921571 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:46.922123 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:46.922139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.922985 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.922988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:46.924013 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:46.924022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:46.924606 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:46.924610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:46.924783 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:46.925465 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:46.925481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:46.927220 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:46.929158 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:46.929165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:46.929181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:46.929506 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:46.929810 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:46.929815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:46.929817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:46.930014 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:46.930420 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:46.930427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:46.930687 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 21:06:46.930872 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 21:06:46.931212 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
I20260928 21:06:46.933276 25450 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37789
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:46.933545 25450 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:46.934118 25450 heartbeater.cc:511] Master 127.21.140.126:37789 requested a full tablet report, sending...
I20260928 21:06:46.935194 25121 ts_manager.cc:194] Registered new tserver with Master: bb9f4340426e485d8979d302ee290a0e (127.21.140.66:39633)
I20260928 21:06:46.935866 25121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:56231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:46.953065 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37789
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:47.067603 25455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:47.067859 25455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:47.067903 25455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:47.071938 25455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:47.072000 25455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:47.072091 25455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:47.077358 25455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:47.078464 25455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:47.079469 25455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:47.089152 25460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:47.089320 25455 server_base.cc:1034] running on GCE node
W20260928 21:06:47.089190 25463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:47.089218 25461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:47.089906 25455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:47.090657 25455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:47.091823 25455 hybrid_clock.cc:648] HybridClock initialized: now 1790629607091771 us; error 68 us; skew 500 ppm
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:47.094668 25455 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:47.095677 25455 webserver.cc:460] Webserver started at http://127.21.140.67:43051/ using document root <none> and password file <none>
I20260928 21:06:47.096316 25455 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:47.096417 25455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:47.096693 25455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:47.098835 25455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "67ebb02e1c9348fbbbf22ea611a88728"
format_stamp: "Formatted at 2026-09-28 21:06:47 on dist-test-slave-d5wc"
server_key: "2c89febb4bfe805d1f9b3ae9a925cf23"
server_key_iv: "5af29b03d06c6c26dc5446fc5a19ab06"
server_key_version: "encryptionkey@0"
I20260928 21:06:47.099408 25455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "67ebb02e1c9348fbbbf22ea611a88728"
format_stamp: "Formatted at 2026-09-28 21:06:47 on dist-test-slave-d5wc"
server_key: "2c89febb4bfe805d1f9b3ae9a925cf23"
server_key_iv: "5af29b03d06c6c26dc5446fc5a19ab06"
server_key_version: "encryptionkey@0"
I20260928 21:06:47.103145 25455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 21:06:47.106127 25470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:47.107582 25455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:47.107708 25455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "67ebb02e1c9348fbbbf22ea611a88728"
format_stamp: "Formatted at 2026-09-28 21:06:47 on dist-test-slave-d5wc"
server_key: "2c89febb4bfe805d1f9b3ae9a925cf23"
server_key_iv: "5af29b03d06c6c26dc5446fc5a19ab06"
server_key_version: "encryptionkey@0"
I20260928 21:06:47.107833 25455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:47.134063 25455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:47.137121 25455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:47.137328 25455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:47.137909 25455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:47.138775 25455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:47.138849 25455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:47.138924 25455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:47.138974 25455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:47.148046 25455 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:34829
I20260928 21:06:47.148064 25583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:34829 every 8 connection(s)
I20260928 21:06:47.149019 25455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:47.150079 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25455
I20260928 21:06:47.150200 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:47.150525 22065 external_mini_cluster.cc:1428] Setting key 06a3d49161d4aa7735b110c3830fe509
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:47.164031 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:47.151244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:49453 (local address 127.21.140.126:37789)
0928 21:06:47.151504 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:47.151508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:47.152464 (+ 956us) server_negotiation.cc:407] Connection header received
0928 21:06:47.153940 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:47.153942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:47.154028 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:47.154123 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:47.155283 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:47.155905 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:47.157155 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:47.157393 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:47.157485 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:47.160288 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:47.160291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:47.160293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:47.160318 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:47.161684 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:47.162367 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:47.162370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:47.162371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:47.162445 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:47.162805 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:47.162808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:47.162809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:47.163010 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:47.163108 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:47.163665 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 21:06:47.163795 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260928 21:06:47.165041 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:47.151711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37789 (local address 127.21.140.67:49453)
0928 21:06:47.152311 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:47.152364 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:47.153693 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:47.154259 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:47.154266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:47.154643 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:47.155134 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:47.155147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:47.156035 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:47.156038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:47.157002 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:47.157011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:47.157592 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:47.157595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:47.157785 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:47.158506 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:47.158523 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:47.160117 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:47.161798 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:47.161803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:47.161816 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:47.162268 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:47.162545 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:47.162548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:47.162550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:47.162695 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:47.163131 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:47.163138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:47.163422 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 21:06:47.163596 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 21:06:47.163983 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":225,"threads_started":1}
I20260928 21:06:47.166265 25584 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37789
I20260928 21:06:47.166476 25584 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:47.166920 25584 heartbeater.cc:511] Master 127.21.140.126:37789 requested a full tablet report, sending...
I20260928 21:06:47.167927 25121 ts_manager.cc:194] Registered new tserver with Master: 67ebb02e1c9348fbbbf22ea611a88728 (127.21.140.67:34829)
I20260928 21:06:47.168550 25121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:49453
I20260928 21:06:47.176849 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629607, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629607, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:47.203290 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:47.194162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.21.140.126:37789)
0928 21:06:47.194319 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:47.194325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:47.194523 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:06:47.194717 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:47.194719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:47.194787 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:47.194888 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:47.195647 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:47.196240 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:47.197097 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:47.197313 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:47.197419 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:47.200100 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:47.200103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:47.200105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:47.200130 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:47.201449 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:47.201909 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:47.201913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:47.201917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:47.202062 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:47.202349 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:47.202353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:47.202355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:47.202593 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:47.202743 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:47.202998 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 21:06:47.203120 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 21:06:47.203749 25121 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44476
I20260928 21:06:47.204769 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25186
I20260928 21:06:47.211181 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25321
I20260928 21:06:47.216761 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25455
I20260928 21:06:47.222133 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25092
2026-09-28T21:06:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5303 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25596](info): set up 2 sockets
Sep 28 21:06:47 dist-test-slave-d5wc krb5kdc[25596](info): commencing operation
krb5kdc: starting...
W20260928 21:06:48.307705 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:48 dist-test-slave-d5wc krb5kdc[25596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:48.441743 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:39897
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:48.553776 25609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:48.554163 25609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:48.554250 25609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:48.558343 25609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:48.558452 25609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:48.558490 25609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:48.558528 25609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:48.558562 25609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:48.564381 25609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:39897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:48.565572 25609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:48.566685 25609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:48.576095 25615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:48.576104 25617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:48.576098 25614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:48.576515 25609 server_base.cc:1034] running on GCE node
I20260928 21:06:48.577167 25609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:48.578559 25609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:48.579777 25609 hybrid_clock.cc:648] HybridClock initialized: now 1790629608579746 us; error 43 us; skew 500 ppm
Sep 28 21:06:48 dist-test-slave-d5wc krb5kdc[25596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629608, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:48.582878 25609 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:48.583950 25609 webserver.cc:460] Webserver started at http://127.21.140.126:46155/ using document root <none> and password file <none>
I20260928 21:06:48.584563 25609 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:48.584661 25609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:48.584892 25609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:48.587111 25609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "66d416ef5eb34db8be3d45995cd4599f"
format_stamp: "Formatted at 2026-09-28 21:06:48 on dist-test-slave-d5wc"
server_key: "d2d60374dba86d6ba371828bfb16bf56"
server_key_iv: "9b71790a92c25f990e45314d55f7556a"
server_key_version: "encryptionkey@0"
I20260928 21:06:48.587761 25609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "66d416ef5eb34db8be3d45995cd4599f"
format_stamp: "Formatted at 2026-09-28 21:06:48 on dist-test-slave-d5wc"
server_key: "d2d60374dba86d6ba371828bfb16bf56"
server_key_iv: "9b71790a92c25f990e45314d55f7556a"
server_key_version: "encryptionkey@0"
I20260928 21:06:48.592770 25609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 21:06:48.595746 25624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:48.597290 25609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:48.597570 25609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "66d416ef5eb34db8be3d45995cd4599f"
format_stamp: "Formatted at 2026-09-28 21:06:48 on dist-test-slave-d5wc"
server_key: "d2d60374dba86d6ba371828bfb16bf56"
server_key_iv: "9b71790a92c25f990e45314d55f7556a"
server_key_version: "encryptionkey@0"
I20260928 21:06:48.597733 25609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:48.631385 25609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:48.634774 25609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:48.634999 25609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:48.642493 25609 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:39897
I20260928 21:06:48.642592 25676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:39897 every 8 connection(s)
I20260928 21:06:48.643652 25609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:48.646561 25677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:48.648904 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25609
I20260928 21:06:48.649016 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:48.649292 22065 external_mini_cluster.cc:1428] Setting key f8fc295ef1824741895ba8a1d13c957c
I20260928 21:06:48.652581 25677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f: Bootstrap starting.
I20260928 21:06:48.654852 25677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:48.655591 25677 log.cc:824] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:48.657603 25677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f: No bootstrap required, opened a new log
Sep 28 21:06:48 dist-test-slave-d5wc krb5kdc[25596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:48.660902 25677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d416ef5eb34db8be3d45995cd4599f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39897 } }
I20260928 21:06:48.661214 25677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:48.661300 25677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d416ef5eb34db8be3d45995cd4599f, State: Initialized, Role: FOLLOWER
I20260928 21:06:48.661782 25677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d416ef5eb34db8be3d45995cd4599f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39897 } }
I20260928 21:06:48.661936 25677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:48.662122 25677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:48.662230 25677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:48.663264 25677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d416ef5eb34db8be3d45995cd4599f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39897 } }
I20260928 21:06:48.663647 25677 leader_election.cc:302] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d416ef5eb34db8be3d45995cd4599f; no voters:
I20260928 21:06:48.663964 25677 leader_election.cc:288] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:48.664073 25682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:48.664350 25682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [term 1 LEADER]: Becoming Leader. State: Replica: 66d416ef5eb34db8be3d45995cd4599f, State: Running, Role: LEADER
I20260928 21:06:48.664719 25682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d416ef5eb34db8be3d45995cd4599f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39897 } }
I20260928 21:06:48.664827 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:48.650875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.21.140.126:39897)
0928 21:06:48.651319 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:48.651334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:48.651369 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:06:48.652012 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:48.652029 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:48.652358 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:48.652708 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:48.653564 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:48.654624 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:48.655539 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:48.656057 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:48.656224 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:48.658850 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:48.658856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:48.658868 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:48.658896 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:48.661437 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:48.661898 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:48.661904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:48.661910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:48.662147 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:48.662468 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:48.662471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:48.662472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:48.662864 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:48.663019 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:48.663406 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 21:06:48.663753 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260928 21:06:48.665305 25677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:48.667678 25683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d416ef5eb34db8be3d45995cd4599f. Latest consensus state: current_term: 1 leader_uuid: "66d416ef5eb34db8be3d45995cd4599f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d416ef5eb34db8be3d45995cd4599f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39897 } } }
I20260928 21:06:48.668246 25683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:48.668045 25684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d416ef5eb34db8be3d45995cd4599f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d416ef5eb34db8be3d45995cd4599f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39897 } } }
I20260928 21:06:48.668466 25684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f [sys.catalog]: This master's current role is: LEADER
W20260928 21:06:48.670047 25696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:06:48.670132 25696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:06:48.670212 25698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:48.670903 25698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:48.676329 25698 catalog_manager.cc:1347] Generated new cluster ID: dd35faee054b4e2db82be59b7ee5d2bc
I20260928 21:06:48.676415 25698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:48.688939 25698 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:48.689886 25698 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:48.695315 25698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d416ef5eb34db8be3d45995cd4599f: Generated new TSK 0
I20260928 21:06:48.696044 25698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:48.733943 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39897
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:48.876364 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:48.876637 25703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:48.876681 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:48.880995 25703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:48.881057 25703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:48.881170 25703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:48.886593 25703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:48.887651 25703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:48.888672 25703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:48.898350 25711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:48.898350 25708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:48.898355 25709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:48.898739 25703 server_base.cc:1034] running on GCE node
I20260928 21:06:48.899185 25703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:48.899894 25703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:48.901104 25703 hybrid_clock.cc:648] HybridClock initialized: now 1790629608901066 us; error 36 us; skew 500 ppm
Sep 28 21:06:48 dist-test-slave-d5wc krb5kdc[25596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629608, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:48.904057 25703 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:48.905019 25703 webserver.cc:460] Webserver started at http://127.21.140.65:43503/ using document root <none> and password file <none>
I20260928 21:06:48.905566 25703 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:48.905630 25703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:48.905799 25703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:48.907929 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "ea8b9cef28474f46874a472192e13b2f"
format_stamp: "Formatted at 2026-09-28 21:06:48 on dist-test-slave-d5wc"
server_key: "3b1c0d39fce4514f3649d3442e5fcd3d"
server_key_iv: "576136e2a28d2426239c37d933116e9f"
server_key_version: "encryptionkey@0"
I20260928 21:06:48.908468 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "ea8b9cef28474f46874a472192e13b2f"
format_stamp: "Formatted at 2026-09-28 21:06:48 on dist-test-slave-d5wc"
server_key: "3b1c0d39fce4514f3649d3442e5fcd3d"
server_key_iv: "576136e2a28d2426239c37d933116e9f"
server_key_version: "encryptionkey@0"
I20260928 21:06:48.912468 25703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:48.915597 25718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:48.917102 25703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:48.917241 25703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "ea8b9cef28474f46874a472192e13b2f"
format_stamp: "Formatted at 2026-09-28 21:06:48 on dist-test-slave-d5wc"
server_key: "3b1c0d39fce4514f3649d3442e5fcd3d"
server_key_iv: "576136e2a28d2426239c37d933116e9f"
server_key_version: "encryptionkey@0"
I20260928 21:06:48.917380 25703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:48.930311 25703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:48.933635 25703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:48.933849 25703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:48.934473 25703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:48.935317 25703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:48.935366 25703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:48.935406 25703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:48.935421 25703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:48.944990 25703 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:42963
I20260928 21:06:48.945020 25831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:42963 every 8 connection(s)
I20260928 21:06:48.945915 25703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:48.951727 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25703
I20260928 21:06:48.951867 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:48.952188 22065 external_mini_cluster.cc:1428] Setting key 11362713d6ce7b651c63f96e0475e717
Sep 28 21:06:48 dist-test-slave-d5wc krb5kdc[25596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629608, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:48.962155 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:48.948067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:49045 (local address 127.21.140.126:39897)
0928 21:06:48.948237 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:48.948241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:48.949221 (+ 980us) server_negotiation.cc:407] Connection header received
0928 21:06:48.950877 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:48.950881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:48.950943 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:48.951033 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:48.952748 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:48.953352 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:48.954852 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:48.955127 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:48.955259 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:48.958164 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:48.958168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:48.958171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:48.958198 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:48.959862 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:48.960474 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:48.960478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:48.960480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:48.960562 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:48.960933 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:48.960941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:48.960942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:48.961168 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:48.961281 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:48.961800 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 21:06:48.961936 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:48.963023 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:48.948474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39897 (local address 127.21.140.65:49045)
0928 21:06:48.949042 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:48.949090 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:48.950621 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:48.951212 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:48.951220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:48.951770 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:48.952552 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:48.952577 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:48.953508 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:48.953511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:48.954702 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:48.954713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:48.955304 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:48.955308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:48.955499 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:48.956187 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:48.956205 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:48.957856 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:48.960050 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:48.960056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:48.960074 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:48.960344 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:48.960659 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:48.960663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:48.960665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:48.960809 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:48.961279 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:48.961286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:48.961553 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 21:06:48.961746 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 21:06:48.962114 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260928 21:06:48.964398 25832 heartbeater.cc:347] Connected to a master server at 127.21.140.126:39897
I20260928 21:06:48.964694 25832 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:48.965279 25832 heartbeater.cc:511] Master 127.21.140.126:39897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:06:48.967259 25636 ts_manager.cc:194] Registered new tserver with Master: ea8b9cef28474f46874a472192e13b2f (127.21.140.65:42963)
I20260928 21:06:48.969107 25636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:49045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:48.986137 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39897
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:49.099572 25837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:49.099865 25837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:49.099910 25837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:49.104362 25837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:49.104430 25837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:49.104491 25837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:49.110108 25837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:49.111240 25837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:49.112291 25837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:49.122270 25842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:49.122282 25843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:49.122283 25845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:49.122622 25837 server_base.cc:1034] running on GCE node
I20260928 21:06:49.123054 25837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:49.123728 25837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:49.124914 25837 hybrid_clock.cc:648] HybridClock initialized: now 1790629609124885 us; error 51 us; skew 500 ppm
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:49.127915 25837 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:49.128865 25837 webserver.cc:460] Webserver started at http://127.21.140.66:35497/ using document root <none> and password file <none>
I20260928 21:06:49.129441 25837 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:49.129534 25837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:49.129781 25837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:49.131906 25837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "0859d801d57b4281bb7067c20f6b8bbe"
format_stamp: "Formatted at 2026-09-28 21:06:49 on dist-test-slave-d5wc"
server_key: "886b91cb488ecf3eec67741fe774a214"
server_key_iv: "ce237f62e64c866dcb0ad7774e84bd48"
server_key_version: "encryptionkey@0"
I20260928 21:06:49.132573 25837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "0859d801d57b4281bb7067c20f6b8bbe"
format_stamp: "Formatted at 2026-09-28 21:06:49 on dist-test-slave-d5wc"
server_key: "886b91cb488ecf3eec67741fe774a214"
server_key_iv: "ce237f62e64c866dcb0ad7774e84bd48"
server_key_version: "encryptionkey@0"
I20260928 21:06:49.136404 25837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 21:06:49.139169 25852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:49.140653 25837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:49.140816 25837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "0859d801d57b4281bb7067c20f6b8bbe"
format_stamp: "Formatted at 2026-09-28 21:06:49 on dist-test-slave-d5wc"
server_key: "886b91cb488ecf3eec67741fe774a214"
server_key_iv: "ce237f62e64c866dcb0ad7774e84bd48"
server_key_version: "encryptionkey@0"
I20260928 21:06:49.140957 25837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:49.166607 25837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:49.170301 25837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:49.170508 25837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:49.171100 25837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:49.171940 25837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:49.172011 25837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:49.172083 25837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:49.172140 25837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:49.181249 25837 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:37825
I20260928 21:06:49.181300 25965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:37825 every 8 connection(s)
I20260928 21:06:49.182287 25837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:49.183368 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25837
I20260928 21:06:49.183493 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:49.183812 22065 external_mini_cluster.cc:1428] Setting key a241bbe162a4e514c64d5e35cd5e883e
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:06:49.197217 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:49.184424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:53063 (local address 127.21.140.126:39897)
0928 21:06:49.184557 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:49.184561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:49.185575 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 21:06:49.187041 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:49.187044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:49.187096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:49.187181 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:49.188362 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.188896 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.190094 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.190362 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.190476 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:49.193032 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:49.193036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:49.193039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:49.193066 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:49.194617 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.195234 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.195237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.195239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.195347 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.195901 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.195905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.195906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.196128 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:49.196274 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:49.196942 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 21:06:49.197086 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:06:49.198767 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:49.184892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39897 (local address 127.21.140.66:53063)
0928 21:06:49.185413 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:49.185468 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:49.186830 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:49.187364 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:49.187371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:49.187739 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:49.188212 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:49.188225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.189025 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.189028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:49.189930 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:49.189939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.190518 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.190522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:49.190713 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:49.191376 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:49.191392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:49.192861 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:49.194753 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:49.194759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:49.194775 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:49.195142 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:49.195535 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:49.195541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:49.195546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:49.195770 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:49.196271 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:49.196280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:49.196595 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 21:06:49.197515 (+ 920us) client_negotiation.cc:240] Negotiation successful
0928 21:06:49.197858 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":163,"threads_started":1}
I20260928 21:06:49.200086 25966 heartbeater.cc:347] Connected to a master server at 127.21.140.126:39897
I20260928 21:06:49.200382 25966 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:49.201104 25966 heartbeater.cc:511] Master 127.21.140.126:39897 requested a full tablet report, sending...
I20260928 21:06:49.202606 25636 ts_manager.cc:194] Registered new tserver with Master: 0859d801d57b4281bb7067c20f6b8bbe (127.21.140.66:37825)
I20260928 21:06:49.203248 25636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:53063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:49.214179 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39897
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:49.332893 25971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:49.333205 25971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:49.333250 25971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:49.337509 25971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:49.337591 25971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:49.337656 25971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:49.343514 25971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:49.344790 25971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:49.346360 25971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:49.358022 25976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:49.358062 25979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:49.358150 25971 server_base.cc:1034] running on GCE node
W20260928 21:06:49.358042 25977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:49.358822 25971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:49.359552 25971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:49.360772 25971 hybrid_clock.cc:648] HybridClock initialized: now 1790629609360752 us; error 36 us; skew 500 ppm
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:49.364012 25971 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:49.365106 25971 webserver.cc:460] Webserver started at http://127.21.140.67:36243/ using document root <none> and password file <none>
I20260928 21:06:49.365684 25971 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:49.365794 25971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:49.366079 25971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:49.368372 25971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "a631a798f0f64400b3768443dd039de0"
format_stamp: "Formatted at 2026-09-28 21:06:49 on dist-test-slave-d5wc"
server_key: "56b867bf406128e03311ec07d2ce9c5c"
server_key_iv: "8ce00aa39d42824d1221ce5d3dc31ba9"
server_key_version: "encryptionkey@0"
I20260928 21:06:49.368950 25971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "a631a798f0f64400b3768443dd039de0"
format_stamp: "Formatted at 2026-09-28 21:06:49 on dist-test-slave-d5wc"
server_key: "56b867bf406128e03311ec07d2ce9c5c"
server_key_iv: "8ce00aa39d42824d1221ce5d3dc31ba9"
server_key_version: "encryptionkey@0"
I20260928 21:06:49.372970 25971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 21:06:49.375662 25986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:49.377636 25971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:49.377787 25971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "a631a798f0f64400b3768443dd039de0"
format_stamp: "Formatted at 2026-09-28 21:06:49 on dist-test-slave-d5wc"
server_key: "56b867bf406128e03311ec07d2ce9c5c"
server_key_iv: "8ce00aa39d42824d1221ce5d3dc31ba9"
server_key_version: "encryptionkey@0"
I20260928 21:06:49.377945 25971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:49.399755 25971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:49.403287 25971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:49.403486 25971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:49.404124 25971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:49.405007 25971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:49.405054 25971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:49.405119 25971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:49.405151 25971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:49.414911 25971 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:40707
I20260928 21:06:49.414942 26099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:40707 every 8 connection(s)
I20260928 21:06:49.415859 25971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:49.421335 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 25971
I20260928 21:06:49.421499 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:49.421876 22065 external_mini_cluster.cc:1428] Setting key 7c924d956a4b02ca193bc62df8e4b676
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:49.430838 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:49.417810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:34727 (local address 127.21.140.126:39897)
0928 21:06:49.418039 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:49.418049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:49.418934 (+ 885us) server_negotiation.cc:407] Connection header received
0928 21:06:49.420563 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:49.420567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:49.420669 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:49.420766 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:49.422306 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.422998 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.424180 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.424408 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.424523 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:49.427137 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:49.427140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:49.427142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:49.427167 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:49.428540 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.429215 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.429219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.429220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.429300 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.429667 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.429672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.429678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.429861 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:49.429974 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:49.430520 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 21:06:49.430662 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 21:06:49.431679 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:49.418263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:39897 (local address 127.21.140.67:34727)
0928 21:06:49.418750 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:49.418798 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:49.420317 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:49.420981 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:49.420990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:49.421450 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:49.422150 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:49.422166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.423131 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.423135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:49.424036 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:49.424046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.424564 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.424568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:49.424749 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:49.425351 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:49.425367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:49.426946 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:49.428779 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:49.428786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:49.428802 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:49.429081 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:49.429404 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:49.429407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:49.429409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:49.429556 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:49.429989 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:49.429996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:49.430266 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 21:06:49.430464 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 21:06:49.430806 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260928 21:06:49.433095 26100 heartbeater.cc:347] Connected to a master server at 127.21.140.126:39897
I20260928 21:06:49.433334 26100 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:49.433811 26100 heartbeater.cc:511] Master 127.21.140.126:39897 requested a full tablet report, sending...
I20260928 21:06:49.434962 25636 ts_manager.cc:194] Registered new tserver with Master: a631a798f0f64400b3768443dd039de0 (127.21.140.67:40707)
I20260928 21:06:49.435734 25636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:34727
I20260928 21:06:49.436224 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:49.464982 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:49.454645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.21.140.65:42963)
0928 21:06:49.454950 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:49.455009 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:49.455034 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:06:49.455088 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:49.455093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:49.455284 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:49.455407 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:49.456069 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.456852 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.457766 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.458231 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.458327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:49.461047 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:49.461057 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:49.461070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:49.461097 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:49.462941 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.463487 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.463492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.463493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.463587 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.463940 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.463950 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.463952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.464261 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:49.464489 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:49.464668 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 21:06:49.464812 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[25596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:49.477197 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:49.468486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.21.140.126:39897)
0928 21:06:49.468667 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:49.468671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:49.468686 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:06:49.468846 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:49.468848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:49.468894 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:49.468982 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:49.469601 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.470263 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.471060 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:49.471281 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:49.471370 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:49.473860 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:49.473864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:49.473866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:49.473894 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:49.475424 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.475898 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.475902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.475904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.475990 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:49.476282 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:49.476286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:49.476288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:49.476523 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:49.476668 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:49.476900 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 21:06:49.477028 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 21:06:49.477618 25636 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56714
I20260928 21:06:49.478314 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25703
I20260928 21:06:49.485109 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25837
I20260928 21:06:49.490700 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25971
I20260928 21:06:49.496553 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 25609
2026-09-28T21:06:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2274 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[26112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[26112](info): set up 2 sockets
Sep 28 21:06:49 dist-test-slave-d5wc krb5kdc[26112](info): commencing operation
krb5kdc: starting...
W20260928 21:06:50.615669 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:50 dist-test-slave-d5wc krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:50.745831 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40489
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:50.856689 26125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:50.857044 26125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:50.857122 26125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:50.861292 26125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:50.861403 26125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:50.861442 26125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:50.861511 26125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:50.861552 26125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:50.867005 26125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:50.868157 26125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:50.869524 26125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:50.878702 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:50.878702 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:50.878703 26133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:50.879061 26125 server_base.cc:1034] running on GCE node
I20260928 21:06:50.879613 26125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:50.880825 26125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:50.882037 26125 hybrid_clock.cc:648] HybridClock initialized: now 1790629610882010 us; error 50 us; skew 500 ppm
Sep 28 21:06:50 dist-test-slave-d5wc krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:50.884968 26125 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:50.885723 26125 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:50.885819 26125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:50.886154 26125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:50.888281 26125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "8a6fc882955c4134a9722db1e2ae747e"
format_stamp: "Formatted at 2026-09-28 21:06:50 on dist-test-slave-d5wc"
server_key: "9d98517c539221782614f719324e6aa9"
server_key_iv: "cbcf145cdfb3a7f99339b25ac17bae46"
server_key_version: "encryptionkey@0"
I20260928 21:06:50.888901 26125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "8a6fc882955c4134a9722db1e2ae747e"
format_stamp: "Formatted at 2026-09-28 21:06:50 on dist-test-slave-d5wc"
server_key: "9d98517c539221782614f719324e6aa9"
server_key_iv: "cbcf145cdfb3a7f99339b25ac17bae46"
server_key_version: "encryptionkey@0"
I20260928 21:06:50.892867 26125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 21:06:50.895588 26140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:50.896974 26125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:50.897146 26125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "8a6fc882955c4134a9722db1e2ae747e"
format_stamp: "Formatted at 2026-09-28 21:06:50 on dist-test-slave-d5wc"
server_key: "9d98517c539221782614f719324e6aa9"
server_key_iv: "cbcf145cdfb3a7f99339b25ac17bae46"
server_key_version: "encryptionkey@0"
I20260928 21:06:50.897279 26125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:50.914866 26125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:50.918273 26125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:50.918490 26125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:50.926060 26125 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40489
I20260928 21:06:50.926391 26192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40489 every 8 connection(s)
I20260928 21:06:50.926801 26125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:50.929757 26193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:50.932683 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26125
I20260928 21:06:50.932842 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:50.933116 22065 external_mini_cluster.cc:1428] Setting key b7b27b5679b80b520c3edd3318644083
I20260928 21:06:50.936043 26193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: Bootstrap starting.
I20260928 21:06:50.938560 26193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:50.939364 26193 log.cc:824] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:50.941387 26193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: No bootstrap required, opened a new log
Sep 28 21:06:50 dist-test-slave-d5wc krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:50.945092 26193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6fc882955c4134a9722db1e2ae747e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40489 } }
I20260928 21:06:50.945364 26193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:50.945426 26193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6fc882955c4134a9722db1e2ae747e, State: Initialized, Role: FOLLOWER
I20260928 21:06:50.945916 26193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6fc882955c4134a9722db1e2ae747e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40489 } }
I20260928 21:06:50.946117 26193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:50.946204 26193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:50.946309 26193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:50.947331 26193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6fc882955c4134a9722db1e2ae747e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40489 } }
I20260928 21:06:50.947708 26193 leader_election.cc:302] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a6fc882955c4134a9722db1e2ae747e; no voters:
I20260928 21:06:50.948058 26193 leader_election.cc:288] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:50.948168 26198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:50.948414 26198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [term 1 LEADER]: Becoming Leader. State: Replica: 8a6fc882955c4134a9722db1e2ae747e, State: Running, Role: LEADER
I20260928 21:06:50.948431 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:50.934662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.21.140.126:40489)
0928 21:06:50.935147 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:50.935164 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:50.935202 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:06:50.935857 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:50.935876 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:50.936299 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:50.936705 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:50.937616 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:50.938814 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:50.939768 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:50.940265 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:50.940455 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:50.943094 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:50.943099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:50.943114 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:50.943142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:50.944923 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:50.945432 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:50.945441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:50.945447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:50.945584 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:50.945891 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:50.945895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:50.945897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:50.946546 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:50.946769 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:50.947061 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 21:06:50.947369 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":135,"threads_started":1}
I20260928 21:06:50.948778 26198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6fc882955c4134a9722db1e2ae747e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40489 } }
I20260928 21:06:50.949391 26193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:50.951452 26200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a6fc882955c4134a9722db1e2ae747e. Latest consensus state: current_term: 1 leader_uuid: "8a6fc882955c4134a9722db1e2ae747e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6fc882955c4134a9722db1e2ae747e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40489 } } }
I20260928 21:06:50.951599 26200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:50.951839 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a6fc882955c4134a9722db1e2ae747e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6fc882955c4134a9722db1e2ae747e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40489 } } }
I20260928 21:06:50.952280 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e [sys.catalog]: This master's current role is: LEADER
W20260928 21:06:50.954571 26212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:06:50.954672 26212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:06:50.954747 26214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:50.955492 26214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:50.961149 26214 catalog_manager.cc:1347] Generated new cluster ID: b4971c85fb8545cb90b2eb62ceb7539f
I20260928 21:06:50.961232 26214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:50.981369 26214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:50.982728 26214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:50.992270 26214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: Generated new TSK 0
I20260928 21:06:50.993173 26214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:51.031005 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40489
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:51.146555 26219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:51.146937 26219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:51.147131 26219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:51.151892 26219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:51.151964 26219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:51.152030 26219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:51.157681 26219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:51.158890 26219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:51.159888 26219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:51.169937 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:51.170226 26219 server_base.cc:1034] running on GCE node
W20260928 21:06:51.170140 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:51.170563 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:51.171007 26219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:51.171603 26219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:51.172808 26219 hybrid_clock.cc:648] HybridClock initialized: now 1790629611172793 us; error 29 us; skew 500 ppm
Sep 28 21:06:51 dist-test-slave-d5wc krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629611, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:51.175820 26219 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:51.176608 26219 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:51.176702 26219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:51.176935 26219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:51.179126 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "74bd1135e49547bb977d43f2236f7086"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "afd0a536cd138ed2bf998ac4639689a3"
server_key_iv: "b657bb0fe213a44b79c13d7bc328f966"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.179692 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "74bd1135e49547bb977d43f2236f7086"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "afd0a536cd138ed2bf998ac4639689a3"
server_key_iv: "b657bb0fe213a44b79c13d7bc328f966"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.183471 26219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:51.186120 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.187530 26219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:06:51.187693 26219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "74bd1135e49547bb977d43f2236f7086"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "afd0a536cd138ed2bf998ac4639689a3"
server_key_iv: "b657bb0fe213a44b79c13d7bc328f966"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.187831 26219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:51.199877 26219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:51.203349 26219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:51.203562 26219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:51.204161 26219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:51.204998 26219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:51.205078 26219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.205169 26219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:51.205221 26219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.214246 26219 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:43815
I20260928 21:06:51.214264 26347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:43815 every 8 connection(s)
I20260928 21:06:51.214839 26219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:51.218231 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26219
I20260928 21:06:51.218354 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:51.218654 22065 external_mini_cluster.cc:1428] Setting key 85fa8f1ce739a4f895b3a0ee49bca389
Sep 28 21:06:51 dist-test-slave-d5wc krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629611, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:51.230279 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.216649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:36795 (local address 127.21.140.126:40489)
0928 21:06:51.216833 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.216838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.218237 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 21:06:51.219324 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.219327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.219380 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.219477 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:51.221095 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.221631 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.222905 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.223143 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.223258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.225851 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:51.225856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:51.225861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:51.225899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:51.227787 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.228351 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.228357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.228360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.228466 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.228923 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.228926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.228927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.229166 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:51.229252 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.229891 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.230138 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 21:06:51.231177 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.217062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40489 (local address 127.21.140.65:36795)
0928 21:06:51.217893 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.217986 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.219141 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.219668 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.219677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.220188 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:51.220898 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.220942 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.221770 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.221777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.222771 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.222780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.223298 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.223302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.223471 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.224152 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:51.224169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:51.225667 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:51.227939 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:51.227944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:51.227960 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:51.228236 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:51.228596 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:51.228600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:51.228603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:51.228818 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:51.229268 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:51.229273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:51.229610 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.229797 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.230160 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:06:51.232566 26348 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40489
I20260928 21:06:51.232856 26348 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:51.233381 26348 heartbeater.cc:511] Master 127.21.140.126:40489 requested a full tablet report, sending...
I20260928 21:06:51.234963 26157 ts_manager.cc:194] Registered new tserver with Master: 74bd1135e49547bb977d43f2236f7086 (127.21.140.65:43815)
I20260928 21:06:51.236109 26157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:36795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:51.249859 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40489
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:51.366015 26353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:51.366317 26353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:51.366367 26353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:51.370440 26353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:51.370497 26353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:51.370558 26353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:51.376025 26353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:51.377462 26353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:51.378911 26353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:51.391458 26358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:51.391541 26362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:51.391745 26359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:51.392102 26353 server_base.cc:1034] running on GCE node
I20260928 21:06:51.392608 26353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:51.393561 26353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:51.394703 26353 hybrid_clock.cc:648] HybridClock initialized: now 1790629611394697 us; error 37 us; skew 500 ppm
Sep 28 21:06:51 dist-test-slave-d5wc krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629611, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:51.397832 26353 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:51.398685 26353 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:51.398782 26353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:51.398998 26353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:51.401337 26353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "f61e48d6ab92409cad8e02f153b629df"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "20edcb9e7b219f4966cb8ec9879a30de"
server_key_iv: "c32a88e4c278e06d37eda82430b88c4d"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.401921 26353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "f61e48d6ab92409cad8e02f153b629df"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "20edcb9e7b219f4966cb8ec9879a30de"
server_key_iv: "c32a88e4c278e06d37eda82430b88c4d"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.405777 26353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:06:51.408543 26368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.410152 26353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:51.410313 26353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "f61e48d6ab92409cad8e02f153b629df"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "20edcb9e7b219f4966cb8ec9879a30de"
server_key_iv: "c32a88e4c278e06d37eda82430b88c4d"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.410446 26353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:51.430316 26353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:51.433595 26353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:51.433817 26353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:51.434468 26353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:51.435328 26353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:51.435400 26353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.435474 26353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:51.435518 26353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.444386 26353 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:38091
I20260928 21:06:51.444401 26481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:38091 every 8 connection(s)
I20260928 21:06:51.444989 26353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:51.447062 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26353
I20260928 21:06:51.447189 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:51.447512 22065 external_mini_cluster.cc:1428] Setting key 0ac7e1b4510bb5634ce1a4e3adb01af4
Sep 28 21:06:51 dist-test-slave-d5wc krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629611, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:51.461673 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.446948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:55197 (local address 127.21.140.126:40489)
0928 21:06:51.447122 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.447129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.448839 (+ 1710us) server_negotiation.cc:407] Connection header received
0928 21:06:51.450378 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.450385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.450455 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.450573 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:51.452492 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.453048 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.454481 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.454709 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.454810 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.457821 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:51.457825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:51.457827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:51.457881 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:51.459505 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.460050 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.460053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.460055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.460142 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.460490 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.460493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.460495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.460689 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:51.460802 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.461354 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.461493 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 21:06:51.462656 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.447428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40489 (local address 127.21.140.66:55197)
0928 21:06:51.448678 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.448738 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.450113 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.450832 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.450846 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.451458 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:51.452150 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.452168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.453190 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.453193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.454341 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.454356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.454885 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.454890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.455139 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.455824 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:51.455848 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:51.457608 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:51.459633 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:51.459642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:51.459659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:51.459933 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:51.460237 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:51.460241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:51.460243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:51.460385 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:51.460798 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:51.460804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:51.461084 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.461314 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.461648 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":164,"threads_started":1}
I20260928 21:06:51.463946 26482 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40489
I20260928 21:06:51.464174 26482 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:51.464607 26482 heartbeater.cc:511] Master 127.21.140.126:40489 requested a full tablet report, sending...
I20260928 21:06:51.465710 26157 ts_manager.cc:194] Registered new tserver with Master: f61e48d6ab92409cad8e02f153b629df (127.21.140.66:38091)
I20260928 21:06:51.466362 26157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:55197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:51.518672 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40489
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:51.630935 26487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:51.631253 26487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:51.631332 26487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:51.635455 26487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:51.635557 26487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:51.635654 26487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:51.640962 26487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:51.642076 26487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:51.643110 26487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:51.652699 26492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:51.652781 26496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:51.652722 26493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:51.652776 26487 server_base.cc:1034] running on GCE node
I20260928 21:06:51.653363 26487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:51.654112 26487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:51.655324 26487 hybrid_clock.cc:648] HybridClock initialized: now 1790629611655307 us; error 27 us; skew 500 ppm
Sep 28 21:06:51 dist-test-slave-d5wc krb5kdc[26112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629611, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:51.658490 26487 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:51.659307 26487 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:51.659413 26487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:51.659674 26487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:51.661901 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "3ab5c84351fe4dfd9425af2bff3e85b8"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "6914e731cc3b3a48832113d8ec70120d"
server_key_iv: "0563416e6fb27461046e8ed8ddd6a2d0"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.662611 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "3ab5c84351fe4dfd9425af2bff3e85b8"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "6914e731cc3b3a48832113d8ec70120d"
server_key_iv: "0563416e6fb27461046e8ed8ddd6a2d0"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.666658 26487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:06:51.669205 26502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.670538 26487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:51.670676 26487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "3ab5c84351fe4dfd9425af2bff3e85b8"
format_stamp: "Formatted at 2026-09-28 21:06:51 on dist-test-slave-d5wc"
server_key: "6914e731cc3b3a48832113d8ec70120d"
server_key_iv: "0563416e6fb27461046e8ed8ddd6a2d0"
server_key_version: "encryptionkey@0"
I20260928 21:06:51.670795 26487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:51.683231 26487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:51.686650 26487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:51.686862 26487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:51.687496 26487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:51.688341 26487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:51.688411 26487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.688484 26487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:51.688536 26487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:51.697723 26487 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:42247
I20260928 21:06:51.697741 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:42247 every 8 connection(s)
I20260928 21:06:51.698406 26487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:51.705180 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26487
I20260928 21:06:51.705318 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:51.705611 22065 external_mini_cluster.cc:1428] Setting key 433ecd1be6111062a90b39f2c65a3827
Sep 28 21:06:51 dist-test-slave-d5wc krb5kdc[26112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629611, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:51.711951 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.700279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:34651 (local address 127.21.140.126:40489)
0928 21:06:51.700422 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.700426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.701658 (+ 1232us) server_negotiation.cc:407] Connection header received
0928 21:06:51.702579 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.702582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.702629 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.702700 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:51.703914 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.704455 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.705956 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.706294 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.706421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.708905 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:51.708908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:51.708910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:51.708935 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:51.710218 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.710676 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.710679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.710680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.710753 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.711044 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.711046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.711047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.711211 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:51.711284 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.711740 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.711834 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:06:51.712800 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.700683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40489 (local address 127.21.140.67:34651)
0928 21:06:51.701510 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.701551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.702399 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.702918 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.702925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.703308 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:51.703774 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.703786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.704564 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.704567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.705813 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.705824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.706457 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.706461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.706643 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.707233 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:51.707249 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:51.708754 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:51.710318 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:51.710322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:51.710334 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:51.710592 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:51.710834 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:51.710837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:51.710838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:51.710966 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:51.711293 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:51.711298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:51.711534 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.711707 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.711954 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260928 21:06:51.714156 26616 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40489
I20260928 21:06:51.714376 26616 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:51.714773 26616 heartbeater.cc:511] Master 127.21.140.126:40489 requested a full tablet report, sending...
I20260928 21:06:51.715873 26157 ts_manager.cc:194] Registered new tserver with Master: 3ab5c84351fe4dfd9425af2bff3e85b8 (127.21.140.67:42247)
I20260928 21:06:51.716482 26157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:34651
I20260928 21:06:51.719370 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:06:51.728852 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.721306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.21.140.126:40489)
0928 21:06:51.721450 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.721456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.721558 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:06:51.721653 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.721655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.721702 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.721769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:51.722563 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.723109 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.723886 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.724105 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.724186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.725449 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:51.725452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:51.725454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:51.725498 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:51.727256 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.727660 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.727663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.727664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.727747 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:51.727989 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:51.727992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:51.727994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:51.728219 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:51.728332 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.728518 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.728643 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:06:51.731849 26157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:51.734905 26157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:51.750029 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.744621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:42247 (local address 127.0.0.1:57212)
0928 21:06:51.745400 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.745414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.745517 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.746170 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.746173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.746190 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:51.746273 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.746279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.747599 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.747602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.748595 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.748601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.749475 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.749478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.749604 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.749628 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.749691 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.749833 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":5,"thread_start_us":92,"threads_started":1}
I20260928 21:06:51.750036 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.744159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.21.140.66:38091)
0928 21:06:51.745109 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.745123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.745147 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:06:51.745226 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.745232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.745451 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.745801 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:51.746055 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.746908 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.748402 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.749243 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.749359 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.749544 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.749763 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.749863 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":118,"threads_started":1}
I20260928 21:06:51.750027 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.743888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:38091 (local address 127.0.0.1:57674)
0928 21:06:51.744650 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.744701 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.744843 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.745619 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.745634 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.745662 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:51.745937 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.745943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.747043 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.747049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.748269 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.748279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.749360 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.749363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.749516 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.749589 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.749730 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.749844 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":153,"spinlock_wait_cycles":15104,"thread_start_us":113,"threads_started":1}
I20260928 21:06:51.750952 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.745015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.21.140.67:42247)
0928 21:06:51.745359 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.745369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.745506 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:06:51.745668 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.745674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.745844 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.746006 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:51.746409 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.747430 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.748713 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.749306 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.750282 (+ 976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.750401 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.750454 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.750618 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":147,"threads_started":1}
I20260928 21:06:51.750955 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.743819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:43815 (local address 127.0.0.1:44166)
0928 21:06:51.744644 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.744696 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.744843 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.745622 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.745627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.745658 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:51.745935 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.745943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.748028 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.748034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.749096 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.749106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.750549 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.750553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.750647 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.750671 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.750745 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.750815 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":51,"spinlock_wait_cycles":11904,"thread_start_us":76,"threads_started":1}
I20260928 21:06:51.751406 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.743932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.21.140.65:43815)
0928 21:06:51.745118 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.745124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.745143 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:06:51.745234 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.745240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.745448 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.745621 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:51.746102 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.747910 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.749247 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.750099 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.750220 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.750363 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.751162 (+ 799us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.751242 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":124,"threads_started":1}
I20260928 21:06:51.753623 26282 tablet_service.cc:1467] Processing CreateTablet for tablet 54bc652fd88b4741909b36fe48f2a26b (DEFAULT_TABLE table=test-table [id=585953936e7242ee9fee7dc7a86f0423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:51.753664 26416 tablet_service.cc:1467] Processing CreateTablet for tablet 54bc652fd88b4741909b36fe48f2a26b (DEFAULT_TABLE table=test-table [id=585953936e7242ee9fee7dc7a86f0423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:51.753659 26550 tablet_service.cc:1467] Processing CreateTablet for tablet 54bc652fd88b4741909b36fe48f2a26b (DEFAULT_TABLE table=test-table [id=585953936e7242ee9fee7dc7a86f0423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:51.754734 26416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bc652fd88b4741909b36fe48f2a26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:51.754740 26550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bc652fd88b4741909b36fe48f2a26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:51.754737 26282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bc652fd88b4741909b36fe48f2a26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:51.761364 26633 tablet_bootstrap.cc:492] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086: Bootstrap starting.
I20260928 21:06:51.762792 26634 tablet_bootstrap.cc:492] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Bootstrap starting.
I20260928 21:06:51.763762 26633 tablet_bootstrap.cc:654] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:51.764521 26633 log.cc:824] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:51.764657 26634 tablet_bootstrap.cc:654] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:51.765937 26634 log.cc:824] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:51.766243 26635 tablet_bootstrap.cc:492] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Bootstrap starting.
I20260928 21:06:51.766912 26633 tablet_bootstrap.cc:492] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086: No bootstrap required, opened a new log
I20260928 21:06:51.767143 26633 ts_tablet_manager.cc:1397] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:06:51.767766 26634 tablet_bootstrap.cc:492] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: No bootstrap required, opened a new log
I20260928 21:06:51.767962 26634 ts_tablet_manager.cc:1397] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:06:51.768476 26635 tablet_bootstrap.cc:654] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:51.769703 26635 log.cc:824] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:51.771201 26634 raft_consensus.cc:348] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.771479 26634 raft_consensus.cc:374] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:51.771528 26634 raft_consensus.cc:729] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e48d6ab92409cad8e02f153b629df, State: Initialized, Role: FOLLOWER
I20260928 21:06:51.771551 26633 raft_consensus.cc:348] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.771725 26635 tablet_bootstrap.cc:492] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: No bootstrap required, opened a new log
I20260928 21:06:51.771770 26633 raft_consensus.cc:374] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:51.771819 26633 raft_consensus.cc:729] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bd1135e49547bb977d43f2236f7086, State: Initialized, Role: FOLLOWER
I20260928 21:06:51.771890 26635 ts_tablet_manager.cc:1397] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:06:51.772202 26634 consensus_queue.cc:260] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.772275 26633 consensus_queue.cc:260] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.773227 26633 ts_tablet_manager.cc:1428] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:06:51.773236 26634 ts_tablet_manager.cc:1428] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:06:51.773339 26348 heartbeater.cc:503] Master 127.21.140.126:40489 was elected leader, sending a full tablet report...
I20260928 21:06:51.774266 26482 heartbeater.cc:503] Master 127.21.140.126:40489 was elected leader, sending a full tablet report...
I20260928 21:06:51.775530 26635 raft_consensus.cc:348] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.775839 26635 raft_consensus.cc:374] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:51.775902 26635 raft_consensus.cc:729] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab5c84351fe4dfd9425af2bff3e85b8, State: Initialized, Role: FOLLOWER
I20260928 21:06:51.776340 26635 consensus_queue.cc:260] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.777334 26635 ts_tablet_manager.cc:1428] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:06:51.777657 26616 heartbeater.cc:503] Master 127.21.140.126:40489 was elected leader, sending a full tablet report...
I20260928 21:06:51.798512 26641 raft_consensus.cc:482] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:51.798694 26641 raft_consensus.cc:504] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.799985 26641 leader_election.cc:288] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bd1135e49547bb977d43f2236f7086 (127.21.140.65:43815), f61e48d6ab92409cad8e02f153b629df (127.21.140.66:38091)
I20260928 21:06:51.804400 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.800374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:35879 (local address 127.21.140.66:38091)
0928 21:06:51.800506 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.800511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.800826 (+ 315us) server_negotiation.cc:407] Connection header received
0928 21:06:51.800994 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.800997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.801047 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.801124 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:51.801403 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.802114 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.803259 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.803720 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.803845 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.803896 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.804160 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.804254 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:06:51.804562 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.800340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:43815 (local address 127.21.140.67:49271)
0928 21:06:51.800800 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.800813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.800943 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.801214 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.801217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.801233 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:51.801373 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.801378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.802311 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.802317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.803180 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.803190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.804143 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.804146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.804247 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.804268 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.804345 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.804415 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":43,"thread_start_us":132,"threads_started":1}
I20260928 21:06:51.804638 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.800503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:49271 (local address 127.21.140.65:43815)
0928 21:06:51.800649 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.800654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.800884 (+ 230us) server_negotiation.cc:407] Connection header received
0928 21:06:51.801051 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.801054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.801107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.801205 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:06:51.801498 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.802195 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.803326 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.803996 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.804152 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.804201 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.804410 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.804497 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:06:51.805045 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.800283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:38091 (local address 127.21.140.67:35879)
0928 21:06:51.800731 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.800744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:51.800871 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:51.801154 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:51.801157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:51.801171 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:06:51.801300 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.801305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.802227 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.802230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.803133 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:51.803143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.803866 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.803870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:51.803993 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.804023 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:06:51.804792 (+ 769us) client_negotiation.cc:240] Negotiation successful
0928 21:06:51.804875 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":34}
I20260928 21:06:51.805249 26302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc652fd88b4741909b36fe48f2a26b" candidate_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd1135e49547bb977d43f2236f7086" is_pre_election: true
I20260928 21:06:51.805619 26302 raft_consensus.cc:2454] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab5c84351fe4dfd9425af2bff3e85b8 in term 0.
I20260928 21:06:51.805810 26436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc652fd88b4741909b36fe48f2a26b" candidate_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e48d6ab92409cad8e02f153b629df" is_pre_election: true
I20260928 21:06:51.806102 26506 leader_election.cc:302] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab5c84351fe4dfd9425af2bff3e85b8, 74bd1135e49547bb977d43f2236f7086; no voters:
I20260928 21:06:51.806174 26436 raft_consensus.cc:2454] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab5c84351fe4dfd9425af2bff3e85b8 in term 0.
I20260928 21:06:51.806362 26641 raft_consensus.cc:2781] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:51.806448 26641 raft_consensus.cc:482] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:51.806509 26641 raft_consensus.cc:3037] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:51.807583 26641 raft_consensus.cc:504] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.807976 26641 leader_election.cc:288] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [CANDIDATE]: Term 1 election: Requested vote from peers 74bd1135e49547bb977d43f2236f7086 (127.21.140.65:43815), f61e48d6ab92409cad8e02f153b629df (127.21.140.66:38091)
I20260928 21:06:51.808301 26302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc652fd88b4741909b36fe48f2a26b" candidate_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd1135e49547bb977d43f2236f7086"
I20260928 21:06:51.808429 26302 raft_consensus.cc:3037] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:51.808432 26436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bc652fd88b4741909b36fe48f2a26b" candidate_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61e48d6ab92409cad8e02f153b629df"
I20260928 21:06:51.808583 26436 raft_consensus.cc:3037] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:51.809502 26302 raft_consensus.cc:2454] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab5c84351fe4dfd9425af2bff3e85b8 in term 1.
I20260928 21:06:51.809681 26436 raft_consensus.cc:2454] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab5c84351fe4dfd9425af2bff3e85b8 in term 1.
I20260928 21:06:51.809829 26506 leader_election.cc:302] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab5c84351fe4dfd9425af2bff3e85b8, 74bd1135e49547bb977d43f2236f7086; no voters:
I20260928 21:06:51.810035 26641 raft_consensus.cc:2781] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:51.810307 26641 raft_consensus.cc:686] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab5c84351fe4dfd9425af2bff3e85b8, State: Running, Role: LEADER
I20260928 21:06:51.810633 26641 consensus_queue.cc:237] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } }
I20260928 21:06:51.813858 26156 catalog_manager.cc:5495] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab5c84351fe4dfd9425af2bff3e85b8 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab5c84351fe4dfd9425af2bff3e85b8" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42247 } health_report { overall_health: HEALTHY } } }
I20260928 21:06:51.828428 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.824638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.21.140.67:42247)
0928 21:06:51.824752 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:51.824756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:51.824770 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:06:51.824911 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:51.824914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:51.824966 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:51.825076 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:06:51.825465 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.826178 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.827108 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:51.827315 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:51.827394 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:51.827650 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:06:51.828105 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:06:51.828195 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:51.828251 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 21:06:51.828300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:06:51.834389 26302 raft_consensus.cc:1261] T 54bc652fd88b4741909b36fe48f2a26b P 74bd1135e49547bb977d43f2236f7086 [term 1 FOLLOWER]: Refusing update from remote peer 3ab5c84351fe4dfd9425af2bff3e85b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:51.834389 26436 raft_consensus.cc:1261] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 1 FOLLOWER]: Refusing update from remote peer 3ab5c84351fe4dfd9425af2bff3e85b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:51.835162 26643 consensus_queue.cc:1035] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bd1135e49547bb977d43f2236f7086" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:51.835369 26641 consensus_queue.cc:1035] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61e48d6ab92409cad8e02f153b629df" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:51.852496 26156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:51.852694 26156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:51.855373 26156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a6fc882955c4134a9722db1e2ae747e: Sending DeleteTablet for 3 replicas of tablet 54bc652fd88b4741909b36fe48f2a26b
I20260928 21:06:51.856233 26416 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bc652fd88b4741909b36fe48f2a26b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:51 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:57674
I20260928 21:06:51.856233 26282 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bc652fd88b4741909b36fe48f2a26b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:51 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:44166
I20260928 21:06:51.856778 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26219
I20260928 21:06:51.856793 26657 tablet_replica.cc:330] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: stopping tablet replica
I20260928 21:06:51.857002 26550 tablet_service.cc:1514] Processing DeleteTablet for tablet 54bc652fd88b4741909b36fe48f2a26b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:51 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:57212
I20260928 21:06:51.857733 26658 tablet_replica.cc:330] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: stopping tablet replica
I20260928 21:06:51.858101 26658 raft_consensus.cc:2229] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:51.858526 26658 raft_consensus.cc:2258] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:51.859390 26657 raft_consensus.cc:2229] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:51.860828 26658 ts_tablet_manager.cc:1905] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:51.861239 26657 raft_consensus.cc:2258] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:51.862743 26657 ts_tablet_manager.cc:1905] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:51.864203 26658 ts_tablet_manager.cc:1918] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:51.864313 26658 log.cc:1196] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/54bc652fd88b4741909b36fe48f2a26b
I20260928 21:06:51.864682 26658 ts_tablet_manager.cc:1939] T 54bc652fd88b4741909b36fe48f2a26b P 3ab5c84351fe4dfd9425af2bff3e85b8: Deleting consensus metadata
I20260928 21:06:51.865923 26144 catalog_manager.cc:4841] TS 3ab5c84351fe4dfd9425af2bff3e85b8 (127.21.140.67:42247): tablet 54bc652fd88b4741909b36fe48f2a26b (table test-table [id=585953936e7242ee9fee7dc7a86f0423]) successfully deleted
I20260928 21:06:51.867043 26657 ts_tablet_manager.cc:1918] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:51.867244 26657 log.cc:1196] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/54bc652fd88b4741909b36fe48f2a26b
W20260928 21:06:51.867447 26144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:43815 (error 108)
I20260928 21:06:51.867709 26657 ts_tablet_manager.cc:1939] T 54bc652fd88b4741909b36fe48f2a26b P f61e48d6ab92409cad8e02f153b629df: Deleting consensus metadata
I20260928 21:06:51.868336 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26353
I20260928 21:06:51.868538 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.868261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:43815 (local address 127.0.0.1:44170)
0928 21:06:51.868376 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.868445 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:06:51.875254 26144 catalog_manager.cc:4584] TS 74bd1135e49547bb977d43f2236f7086 (127.21.140.65:43815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54bc652fd88b4741909b36fe48f2a26b: Network error: Client connection negotiation failed: client connection to 127.21.140.65:43815: connect: Connection refused (error 111)
I20260928 21:06:51.875952 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26487
I20260928 21:06:51.877290 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:51.876255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:38091 (local address 127.0.0.1:57682)
0928 21:06:51.877159 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:51.877216 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":817}
I20260928 21:06:51.882702 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26125
2026-09-28T21:06:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2386 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T21:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:51Z Disabled control of system clock
I20260928 21:06:51.913913 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33565
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:06:52.025076 26663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:52.025359 26663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:52.025396 26663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:52.029755 26663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:52.029836 26663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:52.029851 26663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:52.029868 26663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:52.029884 26663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:52.034780 26663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:33565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:52.035883 26663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:52.037248 26663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:06:52.046738 26663 server_base.cc:1034] running on GCE node
W20260928 21:06:52.046691 26671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.046659 26668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.046674 26669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:52.047586 26663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:52.048780 26663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:52.050034 26663 hybrid_clock.cc:648] HybridClock initialized: now 1790629612049997 us; error 50 us; skew 500 ppm
I20260928 21:06:52.059345 26663 webserver.cc:460] Webserver started at http://127.21.140.126:33355/ using document root <none> and password file <none>
I20260928 21:06:52.060017 26663 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:52.060134 26663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:52.060391 26663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:52.062732 26663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "0f2f173a30e44e15a5dd9d2441d42533"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "26766e96d2dc24bbfaeccfc1c186932e"
server_key_iv: "c62abb0be9936cb2f81c1bb32c0de3fb"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.063365 26663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "0f2f173a30e44e15a5dd9d2441d42533"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "26766e96d2dc24bbfaeccfc1c186932e"
server_key_iv: "c62abb0be9936cb2f81c1bb32c0de3fb"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.067402 26663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:52.070188 26677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.071621 26663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:52.071796 26663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "0f2f173a30e44e15a5dd9d2441d42533"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "26766e96d2dc24bbfaeccfc1c186932e"
server_key_iv: "c62abb0be9936cb2f81c1bb32c0de3fb"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.071938 26663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:52.076848 26663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:52.077504 26663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:52.077708 26663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:52.084895 26663 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:33565
I20260928 21:06:52.084901 26730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:33565 every 8 connection(s)
I20260928 21:06:52.085879 26663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:52.089169 26731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:52.089931 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26663
I20260928 21:06:52.090157 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:52.090468 22065 external_mini_cluster.cc:1428] Setting key 0c5c44bcf8f60e91d0c6e5ebebacb904
I20260928 21:06:52.096192 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.091862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.21.140.126:33565)
0928 21:06:52.092573 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.092589 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.092626 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:06:52.093320 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.093338 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.093710 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.094116 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.094894 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.094898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.094910 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.094920 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.094978 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.094993 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.095189 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.095269 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.095507 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":114,"threads_started":1}
I20260928 21:06:52.097862 26731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533: Bootstrap starting.
I20260928 21:06:52.100136 26731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:52.100801 26731 log.cc:824] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:52.102783 26731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533: No bootstrap required, opened a new log
I20260928 21:06:52.106590 26731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2f173a30e44e15a5dd9d2441d42533" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33565 } }
I20260928 21:06:52.106827 26731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:52.106910 26731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2f173a30e44e15a5dd9d2441d42533, State: Initialized, Role: FOLLOWER
I20260928 21:06:52.107335 26731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2f173a30e44e15a5dd9d2441d42533" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33565 } }
I20260928 21:06:52.107455 26731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:52.107494 26731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:52.107566 26731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:52.108712 26731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2f173a30e44e15a5dd9d2441d42533" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33565 } }
I20260928 21:06:52.109048 26731 leader_election.cc:302] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f2f173a30e44e15a5dd9d2441d42533; no voters:
I20260928 21:06:52.109338 26731 leader_election.cc:288] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:52.109529 26736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:52.109776 26736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2f173a30e44e15a5dd9d2441d42533, State: Running, Role: LEADER
I20260928 21:06:52.110124 26736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2f173a30e44e15a5dd9d2441d42533" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33565 } }
I20260928 21:06:52.110667 26731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:52.111893 26737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f2f173a30e44e15a5dd9d2441d42533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2f173a30e44e15a5dd9d2441d42533" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33565 } } }
I20260928 21:06:52.111892 26738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f2f173a30e44e15a5dd9d2441d42533. Latest consensus state: current_term: 1 leader_uuid: "0f2f173a30e44e15a5dd9d2441d42533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f2f173a30e44e15a5dd9d2441d42533" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33565 } } }
I20260928 21:06:52.112093 26738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:52.112401 26737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:52.112522 26745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:52.115386 26745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:52.121433 26745 catalog_manager.cc:1347] Generated new cluster ID: 30dad437b6cd4871a73088d54be0f03a
I20260928 21:06:52.121515 26745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:52.132479 26745 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:52.132824 26745 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:52.139842 26745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533: Generated new TSK 0
I20260928 21:06:52.140578 26745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:06:52.151580 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33565
--builtin_ntp_servers=127.21.140.84:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:06:52.266774 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:52.267115 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:52.267163 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:52.271301 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:52.271364 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:52.271423 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:52.276482 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:52.277686 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:52.279158 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:52.289059 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.289126 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.289073 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:52.289179 26755 server_base.cc:1034] running on GCE node
I20260928 21:06:52.289822 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:52.290529 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:52.291694 26755 hybrid_clock.cc:648] HybridClock initialized: now 1790629612291661 us; error 52 us; skew 500 ppm
I20260928 21:06:52.293637 26755 webserver.cc:460] Webserver started at http://127.21.140.65:40885/ using document root <none> and password file <none>
I20260928 21:06:52.294310 26755 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:52.294377 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:52.294637 26755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:52.296729 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "66461661d2bfdba284e8bc8f1b563385"
server_key_iv: "9587b42bd0ecb2c251d0e756dbff79cf"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.297310 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "66461661d2bfdba284e8bc8f1b563385"
server_key_iv: "9587b42bd0ecb2c251d0e756dbff79cf"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.301286 26755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 21:06:52.304286 26769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.305699 26755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:52.305799 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "66461661d2bfdba284e8bc8f1b563385"
server_key_iv: "9587b42bd0ecb2c251d0e756dbff79cf"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.306018 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:52.310734 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:52.311458 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:52.311646 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:52.312207 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:52.313158 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:52.313203 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.313265 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:52.313297 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.322120 26755 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:46075
I20260928 21:06:52.322148 26883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:46075 every 8 connection(s)
I20260928 21:06:52.323057 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:52.328235 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26755
I20260928 21:06:52.328377 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:52.328675 22065 external_mini_cluster.cc:1428] Setting key 4c6c3c4bf895f188aec296a5317c19af
I20260928 21:06:52.329800 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.325155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:32853 (local address 127.21.140.126:33565)
0928 21:06:52.325347 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.325353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.326497 (+ 1144us) server_negotiation.cc:407] Connection header received
0928 21:06:52.327833 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.327837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.327877 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.327983 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.329307 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.329311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.329313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.329326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.329350 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.329359 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.329452 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.329630 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.329671 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 21:06:52.330754 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.325493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33565 (local address 127.21.140.65:32853)
0928 21:06:52.326306 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.326350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.327557 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.328167 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.328174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.328266 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.328971 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.328974 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.329040 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.329131 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.329456 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.329464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.329470 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.329647 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.329920 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260928 21:06:52.330758 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33565
--builtin_ntp_servers=127.21.140.84:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:06:52.332129 26884 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33565
I20260928 21:06:52.332373 26884 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:52.332800 26884 heartbeater.cc:511] Master 127.21.140.126:33565 requested a full tablet report, sending...
I20260928 21:06:52.334352 26694 ts_manager.cc:194] Registered new tserver with Master: 18c7c1c0eb0d4d7790fed193e5f99a7e (127.21.140.65:46075)
W20260928 21:06:52.445689 26887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:52.446059 26887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:52.446149 26887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:52.450243 26887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:52.450335 26887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:52.450435 26887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:52.455201 26887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:52.456301 26887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:52.457701 26887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:52.467690 26895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.467686 26893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.467686 26892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:52.467762 26887 server_base.cc:1034] running on GCE node
I20260928 21:06:52.468336 26887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:52.468992 26887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:52.470162 26887 hybrid_clock.cc:648] HybridClock initialized: now 1790629612470146 us; error 28 us; skew 500 ppm
I20260928 21:06:52.471993 26887 webserver.cc:460] Webserver started at http://127.21.140.66:32983/ using document root <none> and password file <none>
I20260928 21:06:52.472582 26887 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:52.472645 26887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:52.472862 26887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:52.475052 26887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "90b1f426c50a4f389356abb394976264"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "2960e7317258bae229463b25624f56de"
server_key_iv: "f61441d38fd6cc83052360224658c49b"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.475672 26887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "90b1f426c50a4f389356abb394976264"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "2960e7317258bae229463b25624f56de"
server_key_iv: "f61441d38fd6cc83052360224658c49b"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.479566 26887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 21:06:52.482118 26901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.483368 26887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:06:52.483520 26887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "90b1f426c50a4f389356abb394976264"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "2960e7317258bae229463b25624f56de"
server_key_iv: "f61441d38fd6cc83052360224658c49b"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.483644 26887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:52.488396 26887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:52.489028 26887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:52.489225 26887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:52.489790 26887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:52.490787 26887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:52.490868 26887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.490937 26887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:52.490988 26887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.500054 26887 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:44361
I20260928 21:06:52.500073 27015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:44361 every 8 connection(s)
I20260928 21:06:52.501048 26887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:52.507509 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.502982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:49503 (local address 127.21.140.126:33565)
0928 21:06:52.503157 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.503162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.504270 (+ 1108us) server_negotiation.cc:407] Connection header received
0928 21:06:52.505620 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.505622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.505662 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.505740 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.506953 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.506955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.506957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.506968 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.506987 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.506994 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.507084 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.507256 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.507337 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:06:52.507601 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 26887
I20260928 21:06:52.507699 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:52.508061 22065 external_mini_cluster.cc:1428] Setting key 034acd1b587290c8036c110f48657cf4
I20260928 21:06:52.508361 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.503458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33565 (local address 127.21.140.66:49503)
0928 21:06:52.504111 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.504162 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.505413 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.505947 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.505954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.506069 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.506698 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.506699 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.506740 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.506809 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.507080 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.507086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.507090 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.507240 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.507437 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260928 21:06:52.509693 27016 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33565
I20260928 21:06:52.509948 27016 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:52.510429 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33565
--builtin_ntp_servers=127.21.140.84:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:06:52.510473 27016 heartbeater.cc:511] Master 127.21.140.126:33565 requested a full tablet report, sending...
I20260928 21:06:52.511690 26694 ts_manager.cc:194] Registered new tserver with Master: 90b1f426c50a4f389356abb394976264 (127.21.140.66:44361)
W20260928 21:06:52.623883 27019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:52.624158 27019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:52.624199 27019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:52.628438 27019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:52.628499 27019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:52.628557 27019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:52.633751 27019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:52.634868 27019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:52.636271 27019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:52.646236 27027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:52.646236 27024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:52.646272 27019 server_base.cc:1034] running on GCE node
W20260928 21:06:52.646286 27025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:52.646947 27019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:52.647522 27019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:52.648693 27019 hybrid_clock.cc:648] HybridClock initialized: now 1790629612648677 us; error 27 us; skew 500 ppm
I20260928 21:06:52.650640 27019 webserver.cc:460] Webserver started at http://127.21.140.67:36855/ using document root <none> and password file <none>
I20260928 21:06:52.651228 27019 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:52.651293 27019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:52.651506 27019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:52.653585 27019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "695cfafc4c234a598915cbb6af723fc6"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "2b806962e760637a388fbe06f1e796eb"
server_key_iv: "3b278a55cc2fd61f75baafa9fca95aef"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.654217 27019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "695cfafc4c234a598915cbb6af723fc6"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "2b806962e760637a388fbe06f1e796eb"
server_key_iv: "3b278a55cc2fd61f75baafa9fca95aef"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.658284 27019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 21:06:52.660766 27033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.662161 27019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:06:52.662302 27019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "695cfafc4c234a598915cbb6af723fc6"
format_stamp: "Formatted at 2026-09-28 21:06:52 on dist-test-slave-d5wc"
server_key: "2b806962e760637a388fbe06f1e796eb"
server_key_iv: "3b278a55cc2fd61f75baafa9fca95aef"
server_key_version: "encryptionkey@0"
I20260928 21:06:52.662421 27019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:52.667048 27019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:52.667708 27019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:52.667897 27019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:52.668444 27019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:52.669365 27019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:52.669435 27019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.669502 27019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:52.669553 27019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:52.678920 27019 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:42537
I20260928 21:06:52.678941 27147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:42537 every 8 connection(s)
I20260928 21:06:52.679924 27019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:52.686244 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.681801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:51209 (local address 127.21.140.126:33565)
0928 21:06:52.681935 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.681939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.683062 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 21:06:52.684389 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.684393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.684436 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.684514 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.685694 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.685697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.685699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.685710 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.685732 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.685739 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.685818 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.686096 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.686139 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:06:52.687263 27150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.682274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33565 (local address 127.21.140.67:51209)
0928 21:06:52.682914 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.682953 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.684169 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.684679 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.684685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.684774 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.685414 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.685416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.685470 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.685538 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.685843 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.685851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.685856 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.686122 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.686366 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260928 21:06:52.687456 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27019
I20260928 21:06:52.687577 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:52.687933 22065 external_mini_cluster.cc:1428] Setting key 01aa4348cd4a495012a5942cdbcdbcc1
I20260928 21:06:52.688542 27148 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33565
I20260928 21:06:52.688789 27148 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:52.689260 27148 heartbeater.cc:511] Master 127.21.140.126:33565 requested a full tablet report, sending...
I20260928 21:06:52.690418 26694 ts_manager.cc:194] Registered new tserver with Master: 695cfafc4c234a598915cbb6af723fc6 (127.21.140.67:42537)
I20260928 21:06:52.701930 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:06:52.706053 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.704157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.21.140.126:33565)
0928 21:06:52.704343 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.704348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.704490 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:06:52.704662 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.704665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.704707 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.704793 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.705513 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.705517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.705520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.705531 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.705552 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.705567 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.705661 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.705791 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.705830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 21:06:52.708855 26694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:06:52.712322 26694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:06:52.724300 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.721184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:42537 (local address 127.0.0.1:51804)
0928 21:06:52.722332 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.722346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.722422 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.722862 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.722867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.722931 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.723506 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.723507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.723524 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.723580 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.723914 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.723923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.723929 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.724049 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.724127 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":82,"thread_start_us":134,"threads_started":1}
I20260928 21:06:52.724529 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.721268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.21.140.67:42537)
0928 21:06:52.722421 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.722435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.722464 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:06:52.722519 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.722524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.722721 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.722837 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.723733 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.723739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.723751 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.723763 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.723805 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.723828 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.723940 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.724304 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.724380 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":137,"threads_started":1}
I20260928 21:06:52.724877 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.721181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:46075 (local address 127.0.0.1:37188)
0928 21:06:52.721983 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.722038 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.722146 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.723832 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.723835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.723850 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.724298 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.724300 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.724311 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.724337 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.724643 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.724646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.724651 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.724717 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.724753 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":94,"threads_started":1}
I20260928 21:06:52.725335 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.721445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.21.140.65:46075)
0928 21:06:52.723484 (+ 2039us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.723507 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.723525 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:06:52.723566 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.723570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.723695 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.723843 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.724453 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.724460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.724472 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.724485 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.724526 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.724549 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.725065 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.725111 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.725194 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1949,"thread_start_us":109,"threads_started":1}
I20260928 21:06:52.727563 27082 tablet_service.cc:1467] Processing CreateTablet for tablet f9a848b9964e42bc88bf7f8e413e720d (DEFAULT_TABLE table=test-table [id=8e2c8f4441fb4fb1924de460a6ad595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:52.727896 26818 tablet_service.cc:1467] Processing CreateTablet for tablet f9a848b9964e42bc88bf7f8e413e720d (DEFAULT_TABLE table=test-table [id=8e2c8f4441fb4fb1924de460a6ad595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:52.728801 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.721934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:44361 (local address 127.0.0.1:54636)
0928 21:06:52.724196 (+ 2262us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.724210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.724874 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.725769 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.725772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.725786 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.726257 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.726259 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.726270 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.726299 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.728543 (+ 2244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.728546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.728550 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.728622 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.728663 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2143,"thread_start_us":95,"threads_started":1}
I20260928 21:06:52.728924 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a848b9964e42bc88bf7f8e413e720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:52.728976 26818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a848b9964e42bc88bf7f8e413e720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:52.729276 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.722133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.21.140.66:44361)
0928 21:06:52.722542 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.722552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.724814 (+ 2262us) server_negotiation.cc:407] Connection header received
0928 21:06:52.725219 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.725225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.725555 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.725696 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.728284 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.728290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.728302 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.728314 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.728359 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.728392 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.728497 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.729061 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.729136 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":119,"threads_started":1}
I20260928 21:06:52.732097 26950 tablet_service.cc:1467] Processing CreateTablet for tablet f9a848b9964e42bc88bf7f8e413e720d (DEFAULT_TABLE table=test-table [id=8e2c8f4441fb4fb1924de460a6ad595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:06:52.733224 26950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a848b9964e42bc88bf7f8e413e720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:52.735977 27165 tablet_bootstrap.cc:492] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: Bootstrap starting.
I20260928 21:06:52.737293 27166 tablet_bootstrap.cc:492] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Bootstrap starting.
I20260928 21:06:52.738883 27165 tablet_bootstrap.cc:654] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:52.739468 27166 tablet_bootstrap.cc:654] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:52.739702 27165 log.cc:824] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:52.740290 27166 log.cc:824] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:52.740494 27168 tablet_bootstrap.cc:492] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Bootstrap starting.
I20260928 21:06:52.742098 27166 tablet_bootstrap.cc:492] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: No bootstrap required, opened a new log
I20260928 21:06:52.742287 27165 tablet_bootstrap.cc:492] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: No bootstrap required, opened a new log
I20260928 21:06:52.742312 27166 ts_tablet_manager.cc:1397] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:06:52.742664 27168 tablet_bootstrap.cc:654] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:52.742818 27165 ts_tablet_manager.cc:1397] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:06:52.743505 27168 log.cc:824] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:52.745574 27168 tablet_bootstrap.cc:492] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: No bootstrap required, opened a new log
I20260928 21:06:52.745800 27168 ts_tablet_manager.cc:1397] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:06:52.746181 27166 raft_consensus.cc:348] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.746423 27166 raft_consensus.cc:374] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:52.746475 27166 raft_consensus.cc:729] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695cfafc4c234a598915cbb6af723fc6, State: Initialized, Role: FOLLOWER
I20260928 21:06:52.746915 27166 consensus_queue.cc:260] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.747237 27165 raft_consensus.cc:348] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.747491 27165 raft_consensus.cc:374] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:52.747565 27165 raft_consensus.cc:729] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c7c1c0eb0d4d7790fed193e5f99a7e, State: Initialized, Role: FOLLOWER
I20260928 21:06:52.747565 27148 heartbeater.cc:503] Master 127.21.140.126:33565 was elected leader, sending a full tablet report...
I20260928 21:06:52.747931 27166 ts_tablet_manager.cc:1428] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:06:52.748034 27165 consensus_queue.cc:260] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.748697 26884 heartbeater.cc:503] Master 127.21.140.126:33565 was elected leader, sending a full tablet report...
I20260928 21:06:52.749269 27165 ts_tablet_manager.cc:1428] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:06:52.750185 27168 raft_consensus.cc:348] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.750447 27168 raft_consensus.cc:374] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:52.750497 27168 raft_consensus.cc:729] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1f426c50a4f389356abb394976264, State: Initialized, Role: FOLLOWER
I20260928 21:06:52.751128 27168 consensus_queue.cc:260] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.752123 27168 ts_tablet_manager.cc:1428] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:06:52.752456 27016 heartbeater.cc:503] Master 127.21.140.126:33565 was elected leader, sending a full tablet report...
W20260928 21:06:52.824316 26885 tablet.cc:2369] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:52.894707 27171 raft_consensus.cc:482] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:06:52.894912 27171 raft_consensus.cc:504] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.896011 27171 leader_election.cc:288] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c7c1c0eb0d4d7790fed193e5f99a7e (127.21.140.65:46075), 90b1f426c50a4f389356abb394976264 (127.21.140.66:44361)
I20260928 21:06:52.898164 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.896365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:33989 (local address 127.21.140.66:44361)
0928 21:06:52.896529 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.896536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.896633 (+ 97us) server_negotiation.cc:407] Connection header received
0928 21:06:52.896764 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.896767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.896805 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.896893 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.897675 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.897679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.897681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.897694 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.897716 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.897726 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.897823 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.897990 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.898031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:06:52.898164 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.896365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:55001 (local address 127.21.140.65:46075)
0928 21:06:52.896684 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.896687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.896700 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:06:52.896763 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.896766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.896805 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.896893 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.897681 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.897684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.897687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.897698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.897716 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.897726 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.897822 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.897990 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.898031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260928 21:06:52.898162 27150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.896274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:44361 (local address 127.21.140.67:33989)
0928 21:06:52.896531 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.896549 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.896657 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.896935 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.896938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.896953 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.897475 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.897477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.897493 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.897526 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.897849 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.897853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.897857 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.897984 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.898049 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 21:06:52.898180 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.896274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:46075 (local address 127.21.140.67:55001)
0928 21:06:52.896610 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.896621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:52.896679 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:52.896935 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:52.896938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:52.896953 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:52.897521 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:06:52.897523 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:52.897530 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:06:52.897548 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:52.897849 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:52.897853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:52.897857 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:06:52.897983 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 21:06:52.898025 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":50,"thread_start_us":107,"threads_started":1}
I20260928 21:06:52.898872 26838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a848b9964e42bc88bf7f8e413e720d" candidate_uuid: "695cfafc4c234a598915cbb6af723fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" is_pre_election: true
I20260928 21:06:52.898872 26970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a848b9964e42bc88bf7f8e413e720d" candidate_uuid: "695cfafc4c234a598915cbb6af723fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b1f426c50a4f389356abb394976264" is_pre_election: true
I20260928 21:06:52.899187 26838 raft_consensus.cc:2454] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 695cfafc4c234a598915cbb6af723fc6 in term 0.
I20260928 21:06:52.899187 26970 raft_consensus.cc:2454] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 695cfafc4c234a598915cbb6af723fc6 in term 0.
I20260928 21:06:52.899688 27035 leader_election.cc:302] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18c7c1c0eb0d4d7790fed193e5f99a7e, 695cfafc4c234a598915cbb6af723fc6, 90b1f426c50a4f389356abb394976264; no voters:
I20260928 21:06:52.899991 27171 raft_consensus.cc:2781] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:06:52.900091 27171 raft_consensus.cc:482] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:06:52.900169 27171 raft_consensus.cc:3037] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:52.901230 27171 raft_consensus.cc:504] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.901590 27171 leader_election.cc:288] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [CANDIDATE]: Term 1 election: Requested vote from peers 18c7c1c0eb0d4d7790fed193e5f99a7e (127.21.140.65:46075), 90b1f426c50a4f389356abb394976264 (127.21.140.66:44361)
I20260928 21:06:52.901947 26838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a848b9964e42bc88bf7f8e413e720d" candidate_uuid: "695cfafc4c234a598915cbb6af723fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e"
I20260928 21:06:52.902010 26970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a848b9964e42bc88bf7f8e413e720d" candidate_uuid: "695cfafc4c234a598915cbb6af723fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b1f426c50a4f389356abb394976264"
I20260928 21:06:52.902148 26838 raft_consensus.cc:3037] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:52.902143 26970 raft_consensus.cc:3037] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:52.903295 26970 raft_consensus.cc:2454] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 695cfafc4c234a598915cbb6af723fc6 in term 1.
I20260928 21:06:52.903322 26838 raft_consensus.cc:2454] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 695cfafc4c234a598915cbb6af723fc6 in term 1.
I20260928 21:06:52.903621 27035 leader_election.cc:302] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 695cfafc4c234a598915cbb6af723fc6, 90b1f426c50a4f389356abb394976264; no voters:
I20260928 21:06:52.903862 27171 raft_consensus.cc:2781] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:52.904100 27171 raft_consensus.cc:686] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 695cfafc4c234a598915cbb6af723fc6, State: Running, Role: LEADER
I20260928 21:06:52.904465 27171 consensus_queue.cc:237] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } }
I20260928 21:06:52.907640 26690 catalog_manager.cc:5495] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 695cfafc4c234a598915cbb6af723fc6 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "695cfafc4c234a598915cbb6af723fc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695cfafc4c234a598915cbb6af723fc6" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 } health_report { overall_health: UNKNOWN } } }
W20260928 21:06:52.930981 27149 tablet.cc:2369] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:06:52.961529 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.959847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.21.140.67:42537)
0928 21:06:52.960049 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:52.960061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:52.960078 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:06:52.960231 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:52.960233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:52.960295 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:52.960393 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:52.961087 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:52.961091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:52.961093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:52.961104 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:52.961130 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:52.961138 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:52.961230 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:52.961341 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 21:06:52.961382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 21:06:52.967730 26970 raft_consensus.cc:1261] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 1 FOLLOWER]: Refusing update from remote peer 695cfafc4c234a598915cbb6af723fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:52.967730 26838 raft_consensus.cc:1261] T f9a848b9964e42bc88bf7f8e413e720d P 18c7c1c0eb0d4d7790fed193e5f99a7e [term 1 FOLLOWER]: Refusing update from remote peer 695cfafc4c234a598915cbb6af723fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:06:52.968566 27175 consensus_queue.cc:1035] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b1f426c50a4f389356abb394976264" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:52.968768 27171 consensus_queue.cc:1035] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c7c1c0eb0d4d7790fed193e5f99a7e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 46075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:06:52.975834 27178 mvcc.cc:205] Tried to move back new op lower bound from 7334418894707535872 to 7334418894459461632. Current Snapshot: MvccSnapshot[applied={T|T < 7334418894707535872}]
I20260928 21:06:52.978742 27180 mvcc.cc:205] Tried to move back new op lower bound from 7334418894707535872 to 7334418894459461632. Current Snapshot: MvccSnapshot[applied={T|T < 7334418894707535872}]
I20260928 21:06:52.980127 27179 mvcc.cc:205] Tried to move back new op lower bound from 7334418894707535872 to 7334418894459461632. Current Snapshot: MvccSnapshot[applied={T|T < 7334418894707535872}]
I20260928 21:06:52.985158 26693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:52.985354 26693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:06:52.988078 26693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f2f173a30e44e15a5dd9d2441d42533: Sending DeleteTablet for 3 replicas of tablet f9a848b9964e42bc88bf7f8e413e720d
I20260928 21:06:52.989073 27082 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a848b9964e42bc88bf7f8e413e720d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:52 UTC) from {username='slave'} at 127.0.0.1:51804
I20260928 21:06:52.989594 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26755
I20260928 21:06:52.989593 27187 tablet_replica.cc:330] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: stopping tablet replica
I20260928 21:06:52.989647 26818 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a848b9964e42bc88bf7f8e413e720d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:52 UTC) from {username='slave'} at 127.0.0.1:37188
I20260928 21:06:52.990543 27187 raft_consensus.cc:2229] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:06:52.991036 27187 raft_consensus.cc:2258] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:52.991779 26950 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a848b9964e42bc88bf7f8e413e720d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:06:52 UTC) from {username='slave'} at 127.0.0.1:54636
I20260928 21:06:52.992333 27188 tablet_replica.cc:330] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: stopping tablet replica
I20260928 21:06:52.992628 27188 raft_consensus.cc:2229] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:06:52.992933 27188 raft_consensus.cc:2258] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:06:52.992929 27187 ts_tablet_manager.cc:1905] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:52.994303 27188 ts_tablet_manager.cc:1905] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:06:52.996306 27187 ts_tablet_manager.cc:1918] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:52.996505 27187 log.cc:1196] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/f9a848b9964e42bc88bf7f8e413e720d
I20260928 21:06:52.996941 27187 ts_tablet_manager.cc:1939] T f9a848b9964e42bc88bf7f8e413e720d P 695cfafc4c234a598915cbb6af723fc6: Deleting consensus metadata
I20260928 21:06:52.997146 27188 ts_tablet_manager.cc:1918] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:06:52.997249 27188 log.cc:1196] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/f9a848b9964e42bc88bf7f8e413e720d
I20260928 21:06:52.997547 27188 ts_tablet_manager.cc:1939] T f9a848b9964e42bc88bf7f8e413e720d P 90b1f426c50a4f389356abb394976264: Deleting consensus metadata
I20260928 21:06:52.998690 26679 catalog_manager.cc:4841] TS 90b1f426c50a4f389356abb394976264 (127.21.140.66:44361): tablet f9a848b9964e42bc88bf7f8e413e720d (table test-table [id=8e2c8f4441fb4fb1924de460a6ad595a]) successfully deleted
I20260928 21:06:52.998692 26680 catalog_manager.cc:4841] TS 695cfafc4c234a598915cbb6af723fc6 (127.21.140.67:42537): tablet f9a848b9964e42bc88bf7f8e413e720d (table test-table [id=8e2c8f4441fb4fb1924de460a6ad595a]) successfully deleted
W20260928 21:06:52.999033 26678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:46075 (error 108)
I20260928 21:06:53.000072 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:52.999769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:46075 (local address 127.0.0.1:37198)
0928 21:06:52.999896 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:52.999983 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:46075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:06:53.000315 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26887
W20260928 21:06:53.000267 26678 catalog_manager.cc:4584] TS 18c7c1c0eb0d4d7790fed193e5f99a7e (127.21.140.65:46075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9a848b9964e42bc88bf7f8e413e720d: Network error: Client connection negotiation failed: client connection to 127.21.140.65:46075: connect: Connection refused (error 111)
I20260928 21:06:53.007021 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27019
I20260928 21:06:53.013314 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 26663
2026-09-28T21:06:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1130 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 21:06:53.036499 22065 mini_oidc.cc:152] Starting JWKS server
I20260928 21:06:53.038245 22065 webserver.cc:460] Webserver started at https://127.0.0.1:34023/ using document root <none> and password file <none>
I20260928 21:06:53.038436 22065 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:06:53.038735 22065 webserver.cc:460] Webserver started at http://127.0.0.1:36149/ using document root <none> and password file <none>
2026-09-28T21:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:53Z Disabled control of system clock
I20260928 21:06:53.048286 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33463
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:33463
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34023/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/testchainca.cert with env {}
W20260928 21:06:53.161878 27195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:53.162223 27195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:53.162266 27195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:53.166451 27195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:53.166528 27195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:53.166544 27195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:53.166563 27195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:53.166579 27195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:06:53.166594 27195 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:06:53.166608 27195 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34023/jwks.json
I20260928 21:06:53.172127 27195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:33463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34023/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:53.173246 27195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:53.174408 27195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:53.183080 27203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:53.183257 27195 server_base.cc:1034] running on GCE node
W20260928 21:06:53.183161 27201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:53.183100 27200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:53.184002 27195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:53.185158 27195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:53.186347 27195 hybrid_clock.cc:648] HybridClock initialized: now 1790629613186333 us; error 28 us; skew 500 ppm
I20260928 21:06:53.188325 27195 webserver.cc:460] Webserver started at http://127.21.140.126:42611/ using document root <none> and password file <none>
I20260928 21:06:53.188917 27195 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:53.189015 27195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:53.189244 27195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:53.191355 27195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "ce98a9c6904848c492b8b4619a0fddda"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "ef8e6d45379ef7e8f2df607824c69ef8"
server_key_iv: "1fea1df6009b7f38c523bf1891e84676"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.191927 27195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "ce98a9c6904848c492b8b4619a0fddda"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "ef8e6d45379ef7e8f2df607824c69ef8"
server_key_iv: "1fea1df6009b7f38c523bf1891e84676"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.195925 27195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:53.198709 27209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:53.200058 27195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:53.200225 27195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "ce98a9c6904848c492b8b4619a0fddda"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "ef8e6d45379ef7e8f2df607824c69ef8"
server_key_iv: "1fea1df6009b7f38c523bf1891e84676"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.200354 27195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:53.214743 27195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:53.215485 27195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:53.215703 27195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:53.223345 27195 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:33463
I20260928 21:06:53.223426 27261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:33463 every 8 connection(s)
I20260928 21:06:53.224654 27195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:53.228252 27262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:53.234117 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda: Bootstrap starting.
I20260928 21:06:53.234678 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27195
I20260928 21:06:53.234815 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:53.235088 22065 external_mini_cluster.cc:1428] Setting key c5a4476f1db4ddc2d8f54a520eecb4d2
I20260928 21:06:53.236901 27262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:53.237668 27262 log.cc:824] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:53.239935 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda: No bootstrap required, opened a new log
I20260928 21:06:53.244521 27262 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce98a9c6904848c492b8b4619a0fddda" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33463 } }
I20260928 21:06:53.244753 27262 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:53.244791 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:53.236476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.21.140.126:33463)
0928 21:06:53.237155 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:53.237170 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:53.237207 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:06:53.237932 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:53.237952 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:53.238437 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:53.238759 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:53.239168 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.240236 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.241427 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.241905 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.242099 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.242911 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:53.242916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:53.242930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:53.243335 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:53.243391 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:53.243410 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:53.243489 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:53.243689 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:06:53.243981 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":144,"threads_started":1}
I20260928 21:06:53.244827 27262 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce98a9c6904848c492b8b4619a0fddda, State: Initialized, Role: FOLLOWER
I20260928 21:06:53.245405 27262 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce98a9c6904848c492b8b4619a0fddda" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33463 } }
I20260928 21:06:53.245543 27262 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:53.245631 27262 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:53.245757 27262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:53.246825 27262 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce98a9c6904848c492b8b4619a0fddda" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33463 } }
I20260928 21:06:53.247243 27262 leader_election.cc:302] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce98a9c6904848c492b8b4619a0fddda; no voters:
I20260928 21:06:53.247623 27262 leader_election.cc:288] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:53.247741 27267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:53.247848 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:06:53.247964 27267 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [term 1 LEADER]: Becoming Leader. State: Replica: ce98a9c6904848c492b8b4619a0fddda, State: Running, Role: LEADER
I20260928 21:06:53.248281 27267 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce98a9c6904848c492b8b4619a0fddda" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33463 } }
I20260928 21:06:53.248917 27262 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:53.249616 27269 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce98a9c6904848c492b8b4619a0fddda. Latest consensus state: current_term: 1 leader_uuid: "ce98a9c6904848c492b8b4619a0fddda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce98a9c6904848c492b8b4619a0fddda" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33463 } } }
I20260928 21:06:53.249747 27269 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:53.249918 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce98a9c6904848c492b8b4619a0fddda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce98a9c6904848c492b8b4619a0fddda" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33463 } } }
I20260928 21:06:53.250064 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:53.250196 27274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:53.252960 27274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:53.258236 27274 catalog_manager.cc:1347] Generated new cluster ID: a5a8a52a227b4f8ea6380ae691fc0e50
I20260928 21:06:53.258312 27274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:53.266546 27274 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:53.267449 27274 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:53.281867 27274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce98a9c6904848c492b8b4619a0fddda: Generated new TSK 0
I20260928 21:06:53.282704 27274 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:06:53.317366 27292 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 21:06:53.317520 27292 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:06:53.318163 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:53.300603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.21.140.126:33463)
0928 21:06:53.300746 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:53.300752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:53.300971 (+ 219us) server_negotiation.cc:407] Connection header received
0928 21:06:53.301204 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:53.301207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:53.301253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:53.301336 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:06:53.301635 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.302250 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.303471 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.303697 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.303829 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.304206 (+ 377us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:06:53.317685 (+ 13479us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:06:53.318015 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 21:06:53.318276 27265 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 21:06:53.319730 27291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:53.300521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33463 (local address 127.0.0.1:48278)
0928 21:06:53.300866 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:53.300882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:53.301028 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:53.301381 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:53.301385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:53.301393 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:06:53.301528 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:53.301535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.302355 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.302359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:53.303338 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:53.303349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.303848 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.303852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:53.303950 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.304010 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:06:53.317991 (+ 13981us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:06:53.318299 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.140.126:33463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.140.126:33463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 21:06:53.320816 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27195
2026-09-28T21:06:53Z chronyd exiting
I20260928 21:06:53.357957 22065 test_util.cc:175] -----------------------------------------------
I20260928 21:06:53.358177 22065 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629583469051-22065-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 21:06:53.359313 22065 mini_oidc.cc:152] Starting JWKS server
I20260928 21:06:53.360219 22065 webserver.cc:460] Webserver started at https://127.0.0.1:43411/ using document root <none> and password file <none>
I20260928 21:06:53.360368 22065 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:06:53.360682 22065 webserver.cc:460] Webserver started at http://127.0.0.1:41227/ using document root <none> and password file <none>
2026-09-28T21:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:53Z Disabled control of system clock
I20260928 21:06:53.370188 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:43713
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:46215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:43713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43411/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/testchainca.cert with env {}
W20260928 21:06:53.490242 27298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:53.490557 27298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:53.490600 27298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:53.494738 27298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:53.494809 27298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:53.494824 27298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:53.494843 27298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:53.494858 27298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:06:53.494874 27298 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:06:53.494887 27298 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43411/jwks.json
I20260928 21:06:53.500458 27298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:43713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:43713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43411/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:53.501549 27298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:53.502552 27298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:53.511202 27306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:53.511305 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:53.511464 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:53.511552 27298 server_base.cc:1034] running on GCE node
I20260928 21:06:53.512153 27298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:53.513343 27298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:53.514519 27298 hybrid_clock.cc:648] HybridClock initialized: now 1790629613514459 us; error 72 us; skew 500 ppm
I20260928 21:06:53.516726 27298 webserver.cc:460] Webserver started at http://127.21.140.126:34615/ using document root <none> and password file <none>
I20260928 21:06:53.517398 27298 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:53.517470 27298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:53.517714 27298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:53.520519 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "7e422dc70caf4c019e0bdf902c3bc3d4"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "cebeb276a835e6f4a7ec09b69c3b53aa"
server_key_iv: "c41704b15d3235393cf0db46317815a6"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.521298 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "7e422dc70caf4c019e0bdf902c3bc3d4"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "cebeb276a835e6f4a7ec09b69c3b53aa"
server_key_iv: "c41704b15d3235393cf0db46317815a6"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.525544 27298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:06:53.528431 27312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:53.529781 27298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:53.529934 27298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "7e422dc70caf4c019e0bdf902c3bc3d4"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "cebeb276a835e6f4a7ec09b69c3b53aa"
server_key_iv: "c41704b15d3235393cf0db46317815a6"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.530084 27298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:53.540303 27298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:53.540977 27298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:53.541175 27298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:53.548327 27298 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:43713
I20260928 21:06:53.548336 27364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:43713 every 8 connection(s)
I20260928 21:06:53.549319 27298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:53.552683 27365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:53.556788 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27298
I20260928 21:06:53.556888 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:53.557191 22065 external_mini_cluster.cc:1428] Setting key e494985c821fccde8dc6239cb6117980
I20260928 21:06:53.558681 27365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4: Bootstrap starting.
I20260928 21:06:53.561096 27365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:53.561864 27365 log.cc:824] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:53.563802 27365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4: No bootstrap required, opened a new log
I20260928 21:06:53.566558 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:53.558605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.21.140.126:43713)
0928 21:06:53.559238 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:53.559252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:53.559289 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:06:53.559922 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:53.559939 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:53.560441 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:53.560793 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:53.561334 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.562396 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.563287 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.563768 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.563941 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.564792 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:53.564797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:53.564810 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:53.565200 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:53.565254 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:53.565273 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:53.565384 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:53.565500 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:06:53.565762 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260928 21:06:53.567507 27365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43713 } }
I20260928 21:06:53.567759 27365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:53.567838 27365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e422dc70caf4c019e0bdf902c3bc3d4, State: Initialized, Role: FOLLOWER
I20260928 21:06:53.568272 27365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43713 } }
I20260928 21:06:53.568441 27365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:53.568517 27365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:53.568617 27365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:53.569397 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:06:53.569706 27365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43713 } }
I20260928 21:06:53.570107 27365 leader_election.cc:302] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e422dc70caf4c019e0bdf902c3bc3d4; no voters:
I20260928 21:06:53.570478 27365 leader_election.cc:288] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:53.570557 27370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:53.570760 27370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e422dc70caf4c019e0bdf902c3bc3d4, State: Running, Role: LEADER
I20260928 21:06:53.571066 27370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43713 } }
I20260928 21:06:53.571805 27365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:53.572860 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e422dc70caf4c019e0bdf902c3bc3d4. Latest consensus state: current_term: 1 leader_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43713 } } }
I20260928 21:06:53.573004 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:53.573263 27371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e422dc70caf4c019e0bdf902c3bc3d4" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43713 } } }
I20260928 21:06:53.573354 27371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:53.573434 27379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:53.576732 27379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:53.582330 27379 catalog_manager.cc:1347] Generated new cluster ID: f48045f831614d1e91b3e0ad67dfebae
I20260928 21:06:53.582403 27379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:53.608409 27379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:53.609751 27379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:53.617192 27379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e422dc70caf4c019e0bdf902c3bc3d4: Generated new TSK 0
I20260928 21:06:53.617938 27379 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:06:53.649569 27395 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 21:06:53.649676 27395 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:06:53.650465 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:53.633951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.21.140.126:43713)
0928 21:06:53.634112 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:53.634117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:53.634328 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:06:53.634485 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:53.634489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:53.634550 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:53.634686 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:06:53.634895 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.635793 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.636746 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.637104 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.637215 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.637568 (+ 353us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:06:53.650032 (+ 12464us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:06:53.650319 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 21:06:53.650537 27368 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 21:06:53.650606 27394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:53.633854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43713 (local address 127.0.0.1:36058)
0928 21:06:53.634230 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:53.634243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:53.634347 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:53.634687 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:53.634690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:53.634696 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:06:53.634792 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:53.634797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.635943 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.635946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:53.636620 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:53.636629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.637272 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.637276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:53.637351 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.637379 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:06:53.650262 (+ 12883us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:06:53.650418 (+ 156us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.140.126:43713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":154,"threads_started":1}
I20260928 21:06:53.651376 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27298
2026-09-28T21:06:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 21:06:53.691341 22065 mini_oidc.cc:152] Starting JWKS server
I20260928 21:06:53.692261 22065 webserver.cc:460] Webserver started at https://127.0.0.1:41819/ using document root <none> and password file <none>
I20260928 21:06:53.692417 22065 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:06:53.692735 22065 webserver.cc:460] Webserver started at http://127.0.0.1:36055/ using document root <none> and password file <none>
2026-09-28T21:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:53Z Disabled control of system clock
I20260928 21:06:53.701915 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40617
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40617
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41819/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:06:53.813719 27401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:53.814101 27401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:53.814180 27401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:53.818279 27401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:53.818379 27401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:53.818419 27401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:53.818454 27401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:53.818485 27401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:06:53.818521 27401 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:06:53.818552 27401 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41819/jwks.json
I20260928 21:06:53.823542 27401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41819/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:53.824684 27401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:53.826165 27401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:53.834987 27407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:53.834985 27409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:53.835072 27406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:53.835279 27401 server_base.cc:1034] running on GCE node
I20260928 21:06:53.836073 27401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:53.837198 27401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:53.838375 27401 hybrid_clock.cc:648] HybridClock initialized: now 1790629613838357 us; error 32 us; skew 500 ppm
I20260928 21:06:53.840372 27401 webserver.cc:460] Webserver started at http://127.21.140.126:35199/ using document root <none> and password file <none>
I20260928 21:06:53.840988 27401 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:53.841053 27401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:53.841280 27401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:53.843429 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "99899f8aec75499a82d392a9a32281b6"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "104ee87e2c26106a9c5a353f03c58ace"
server_key_iv: "e92eddce3185f02857da65282f8587c2"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.844084 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "99899f8aec75499a82d392a9a32281b6"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "104ee87e2c26106a9c5a353f03c58ace"
server_key_iv: "e92eddce3185f02857da65282f8587c2"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.848114 27401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:53.850963 27415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:53.852399 27401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:53.852586 27401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "99899f8aec75499a82d392a9a32281b6"
format_stamp: "Formatted at 2026-09-28 21:06:53 on dist-test-slave-d5wc"
server_key: "104ee87e2c26106a9c5a353f03c58ace"
server_key_iv: "e92eddce3185f02857da65282f8587c2"
server_key_version: "encryptionkey@0"
I20260928 21:06:53.852725 27401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:53.869642 27401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:53.870520 27401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:53.870754 27401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:53.878314 27401 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40617
I20260928 21:06:53.878338 27467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40617 every 8 connection(s)
I20260928 21:06:53.879388 27401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:53.883504 27468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:53.887805 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27401
I20260928 21:06:53.887912 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:53.888221 22065 external_mini_cluster.cc:1428] Setting key 3a64c254060c3a40b6701f1529efa0e4
I20260928 21:06:53.889446 27468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6: Bootstrap starting.
I20260928 21:06:53.892269 27468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:53.893158 27468 log.cc:824] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:53.895406 27468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6: No bootstrap required, opened a new log
I20260928 21:06:53.899006 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:53.889662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.21.140.126:40617)
0928 21:06:53.890366 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:53.890387 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:53.890429 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:06:53.891313 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:53.891335 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:53.891847 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:53.892226 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:53.892722 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.894103 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.895038 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.895742 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.895907 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.896749 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:53.896755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:53.896772 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:53.897284 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:53.897359 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:53.897384 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:53.897488 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:53.897619 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:06:53.897912 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260928 21:06:53.900056 27468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99899f8aec75499a82d392a9a32281b6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40617 } }
I20260928 21:06:53.900315 27468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:53.900396 27468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99899f8aec75499a82d392a9a32281b6, State: Initialized, Role: FOLLOWER
I20260928 21:06:53.900882 27468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99899f8aec75499a82d392a9a32281b6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40617 } }
I20260928 21:06:53.901099 27468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:53.901170 27468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:53.901262 27468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:53.902217 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:06:53.902453 27468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99899f8aec75499a82d392a9a32281b6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40617 } }
I20260928 21:06:53.902794 27468 leader_election.cc:302] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99899f8aec75499a82d392a9a32281b6; no voters:
I20260928 21:06:53.903131 27468 leader_election.cc:288] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:53.903208 27473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:53.903406 27473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [term 1 LEADER]: Becoming Leader. State: Replica: 99899f8aec75499a82d392a9a32281b6, State: Running, Role: LEADER
I20260928 21:06:53.903710 27473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99899f8aec75499a82d392a9a32281b6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40617 } }
I20260928 21:06:53.904443 27468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:53.905534 27474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99899f8aec75499a82d392a9a32281b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99899f8aec75499a82d392a9a32281b6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40617 } } }
I20260928 21:06:53.905676 27474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:53.905947 27475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99899f8aec75499a82d392a9a32281b6. Latest consensus state: current_term: 1 leader_uuid: "99899f8aec75499a82d392a9a32281b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99899f8aec75499a82d392a9a32281b6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40617 } } }
I20260928 21:06:53.906069 27475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:53.906219 27482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:53.909427 27482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:53.915768 27482 catalog_manager.cc:1347] Generated new cluster ID: 2df6ad61a36342929b1b87a69830b7ee
I20260928 21:06:53.915864 27482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:53.935271 27482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:53.936599 27482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:53.944453 27482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99899f8aec75499a82d392a9a32281b6: Generated new TSK 0
I20260928 21:06:53.945255 27482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:06:53.970423 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:53.964351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.21.140.126:40617)
0928 21:06:53.964500 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:53.964506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:53.964675 (+ 169us) server_negotiation.cc:407] Connection header received
0928 21:06:53.964832 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:53.964834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:53.964882 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:53.964976 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:06:53.965243 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.965773 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.967046 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.967279 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.967365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.967719 (+ 354us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:06:53.969944 (+ 2225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:06:53.970288 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39986: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 21:06:53.970515 27471 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39986: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 21:06:53.973596 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:53.964268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40617 (local address 127.0.0.1:39986)
0928 21:06:53.964577 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:53.964593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:53.964709 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:53.965007 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:53.965010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:53.965017 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:06:53.965140 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:53.965146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.965886 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.965889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:53.966914 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:53.966925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:53.967413 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:53.967417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:53.967513 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:53.967546 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:06:53.970197 (+ 2651us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:06:53.973416 (+ 3219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.140.126:40617: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41819/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":3000,"thread_start_us":106,"threads_started":1}
I20260928 21:06:53.974093 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27401
E20260928 21:06:53.982957 27498 webserver.cc:533] Webserver: sslize failed:
E20260928 21:06:53.983042 27498 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T21:06:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T21:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:54Z Disabled control of system clock
I20260928 21:06:54.022641 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:43679
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:43679
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/testchainca.cert with env {}
W20260928 21:06:54.168463 27502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:54.168913 27502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:54.168965 27502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:54.173672 27502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:54.173825 27502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:54.173844 27502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:54.173867 27502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:54.173887 27502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:06:54.173908 27502 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:06:54.173925 27502 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 21:06:54.181279 27502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:43679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:54.183041 27502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:54.184540 27502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:54.197113 27508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:54.197113 27507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:54.197113 27510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:54.197741 27502 server_base.cc:1034] running on GCE node
I20260928 21:06:54.198652 27502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:54.200307 27502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:54.201526 27502 hybrid_clock.cc:648] HybridClock initialized: now 1790629614201510 us; error 43 us; skew 500 ppm
I20260928 21:06:54.204408 27502 webserver.cc:460] Webserver started at http://127.21.140.126:44967/ using document root <none> and password file <none>
I20260928 21:06:54.205265 27502 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:54.205356 27502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:54.205659 27502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:54.208675 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "ea1bf5cc13e24803a622b7686869b4b7"
format_stamp: "Formatted at 2026-09-28 21:06:54 on dist-test-slave-d5wc"
server_key: "d5305fe01356539eaa2326ca8b0911a9"
server_key_iv: "b676a1cf29f458c8665814d1546ea8c8"
server_key_version: "encryptionkey@0"
I20260928 21:06:54.209476 27502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "ea1bf5cc13e24803a622b7686869b4b7"
format_stamp: "Formatted at 2026-09-28 21:06:54 on dist-test-slave-d5wc"
server_key: "d5305fe01356539eaa2326ca8b0911a9"
server_key_iv: "b676a1cf29f458c8665814d1546ea8c8"
server_key_version: "encryptionkey@0"
I20260928 21:06:54.214946 27502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 21:06:54.218675 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:54.220602 27502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:54.220832 27502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "ea1bf5cc13e24803a622b7686869b4b7"
format_stamp: "Formatted at 2026-09-28 21:06:54 on dist-test-slave-d5wc"
server_key: "d5305fe01356539eaa2326ca8b0911a9"
server_key_iv: "b676a1cf29f458c8665814d1546ea8c8"
server_key_version: "encryptionkey@0"
I20260928 21:06:54.221029 27502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:54.244782 27502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:54.245918 27502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:54.246315 27502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:54.258445 27502 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:43679
I20260928 21:06:54.258502 27568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:43679 every 8 connection(s)
I20260928 21:06:54.260236 27502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:54.264119 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27502
I20260928 21:06:54.264324 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:54.264693 22065 external_mini_cluster.cc:1428] Setting key ff1a75ca397c79b480090ce0a1233b83
I20260928 21:06:54.265201 27569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:54.275581 27569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7: Bootstrap starting.
I20260928 21:06:54.279496 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:54.266719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.21.140.126:43679)
0928 21:06:54.267756 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:54.267782 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:54.267843 (+ 61us) server_negotiation.cc:407] Connection header received
0928 21:06:54.269048 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:54.269089 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:54.269920 (+ 831us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:54.270530 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:54.271163 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:54.272830 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:54.274114 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:54.275004 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:54.275214 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:54.276455 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:54.276464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:54.276489 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:06:54.277179 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:54.277299 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:06:54.277340 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:54.277478 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:54.277716 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:06:54.278166 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":186,"threads_started":1}
I20260928 21:06:54.279596 27569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:54.280622 27569 log.cc:824] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:54.283830 27569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7: No bootstrap required, opened a new log
I20260928 21:06:54.284068 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:06:54.288718 27569 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1bf5cc13e24803a622b7686869b4b7" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43679 } }
I20260928 21:06:54.289124 27569 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:54.289168 27569 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1bf5cc13e24803a622b7686869b4b7, State: Initialized, Role: FOLLOWER
I20260928 21:06:54.289719 27569 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1bf5cc13e24803a622b7686869b4b7" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43679 } }
I20260928 21:06:54.289860 27569 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:54.289911 27569 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:54.290105 27569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:54.291692 27569 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1bf5cc13e24803a622b7686869b4b7" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43679 } }
I20260928 21:06:54.292224 27569 leader_election.cc:302] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea1bf5cc13e24803a622b7686869b4b7; no voters:
I20260928 21:06:54.292712 27569 leader_election.cc:288] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:54.292869 27574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:54.293164 27574 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [term 1 LEADER]: Becoming Leader. State: Replica: ea1bf5cc13e24803a622b7686869b4b7, State: Running, Role: LEADER
I20260928 21:06:54.293617 27574 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1bf5cc13e24803a622b7686869b4b7" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43679 } }
I20260928 21:06:54.294621 27569 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:54.296398 27576 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1bf5cc13e24803a622b7686869b4b7. Latest consensus state: current_term: 1 leader_uuid: "ea1bf5cc13e24803a622b7686869b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1bf5cc13e24803a622b7686869b4b7" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43679 } } }
I20260928 21:06:54.296409 27575 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea1bf5cc13e24803a622b7686869b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1bf5cc13e24803a622b7686869b4b7" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 43679 } } }
I20260928 21:06:54.296595 27576 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:54.296594 27575 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:54.297118 27584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:54.301442 27584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:54.309478 27584 catalog_manager.cc:1347] Generated new cluster ID: 9965854df851465c898624173cca4e2d
I20260928 21:06:54.309617 27584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:54.326395 27584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:54.327788 27584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:54.333912 27584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea1bf5cc13e24803a622b7686869b4b7: Generated new TSK 0
I20260928 21:06:54.334985 27584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:06:54.784907 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:54.347874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.21.140.126:43679)
0928 21:06:54.348084 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:54.348090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:54.348277 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:06:54.348500 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:54.348505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:54.348599 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:54.348768 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:06:54.349040 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:54.350093 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:54.351127 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:54.351553 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:54.351700 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:54.352211 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:06:54.784322 (+432111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:06:54.784736 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53868: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":102}
W20260928 21:06:54.785019 27572 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53868: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 21:06:54.785166 27598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:54.347738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:43679 (local address 127.0.0.1:53868)
0928 21:06:54.348150 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:54.348165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:54.348303 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:54.348762 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:54.348764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:54.348772 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:06:54.348907 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:54.348913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:54.350240 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:54.350244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:54.350964 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:54.350974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:54.351793 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:54.351797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:54.351897 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:54.351934 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:06:54.784652 (+432718us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:06:54.784906 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.140.126:43679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":147,"threads_started":1}
I20260928 21:06:54.785919 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27502
2026-09-28T21:06:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (794 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (172 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (150 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:55 dist-test-slave-d5wc krb5kdc[27610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:55 dist-test-slave-d5wc krb5kdc[27610](info): set up 2 sockets
Sep 28 21:06:55 dist-test-slave-d5wc krb5kdc[27610](info): commencing operation
krb5kdc: starting...
W20260928 21:06:56.207864 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:56 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:06:56.337574 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34263
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:34263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:56.453060 27623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:56.453377 27623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:56.453460 27623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:56.457475 27623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:56.457573 27623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:56.457613 27623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:56.457672 27623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:56.457711 27623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:56.463054 27623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:34263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:56.464154 27623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:56.465215 27623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:56.474706 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:56.474714 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:56.474788 27623 server_base.cc:1034] running on GCE node
W20260928 21:06:56.474714 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:56.475445 27623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:56.476684 27623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:56.477918 27623 hybrid_clock.cc:648] HybridClock initialized: now 1790629616477885 us; error 45 us; skew 500 ppm
Sep 28 21:06:56 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629616, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:56.480939 27623 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:06:56.481876 27623 webserver.cc:460] Webserver started at http://127.21.140.126:45261/ using document root <none> and password file <none>
I20260928 21:06:56.482509 27623 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:56.482609 27623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:56.482853 27623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:56.484941 27623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "3a6b6b76e75a4b9f8f7532c86997aae3"
format_stamp: "Formatted at 2026-09-28 21:06:56 on dist-test-slave-d5wc"
server_key: "c6955fa9f4082cf8be1eddfbdeb77066"
server_key_iv: "49640b40542e6cde229c36eaa7bc5dad"
server_key_version: "encryptionkey@0"
I20260928 21:06:56.485545 27623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "3a6b6b76e75a4b9f8f7532c86997aae3"
format_stamp: "Formatted at 2026-09-28 21:06:56 on dist-test-slave-d5wc"
server_key: "c6955fa9f4082cf8be1eddfbdeb77066"
server_key_iv: "49640b40542e6cde229c36eaa7bc5dad"
server_key_version: "encryptionkey@0"
I20260928 21:06:56.489603 27623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 21:06:56.492274 27638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:56.493587 27623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:06:56.493742 27623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "3a6b6b76e75a4b9f8f7532c86997aae3"
format_stamp: "Formatted at 2026-09-28 21:06:56 on dist-test-slave-d5wc"
server_key: "c6955fa9f4082cf8be1eddfbdeb77066"
server_key_iv: "49640b40542e6cde229c36eaa7bc5dad"
server_key_version: "encryptionkey@0"
I20260928 21:06:56.493866 27623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:56.524633 27623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:56.527966 27623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:56.528203 27623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:56.536053 27623 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:34263
I20260928 21:06:56.536073 27690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:34263 every 8 connection(s)
I20260928 21:06:56.537137 27623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:56.540330 27691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:06:56.544361 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27623
I20260928 21:06:56.544528 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:56.544826 22065 external_mini_cluster.cc:1428] Setting key ecbf7583de2206d29434f7d1f49d5a4c
I20260928 21:06:56.546454 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3: Bootstrap starting.
I20260928 21:06:56.548784 27691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:56.549574 27691 log.cc:824] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:56.551473 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3: No bootstrap required, opened a new log
Sep 28 21:06:56 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:56.556492 27691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34263 } }
I20260928 21:06:56.556802 27691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:56.556905 27691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6b6b76e75a4b9f8f7532c86997aae3, State: Initialized, Role: FOLLOWER
I20260928 21:06:56.557452 27691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34263 } }
I20260928 21:06:56.557629 27691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:56.557708 27691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:56.557814 27691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:56.559031 27691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34263 } }
I20260928 21:06:56.559402 27691 leader_election.cc:302] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a6b6b76e75a4b9f8f7532c86997aae3; no voters:
I20260928 21:06:56.559736 27691 leader_election.cc:288] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:56.559852 27696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:56.560053 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:56.546307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.21.140.126:34263)
0928 21:06:56.546782 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:56.546798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:56.546834 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:06:56.547509 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:56.547525 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:56.547873 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:56.548221 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:56.549098 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:56.550191 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:56.551045 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:56.551523 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:56.551687 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:56.554516 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:56.554524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:56.554546 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:56.554585 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:56.556635 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:56.557199 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:56.557206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:56.557211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:56.557347 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:56.557664 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:56.557668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:56.557670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:56.558236 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:56.558458 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:56.558678 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 21:06:56.559011 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":144,"threads_started":1}
I20260928 21:06:56.560082 27696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6b6b76e75a4b9f8f7532c86997aae3, State: Running, Role: LEADER
I20260928 21:06:56.560401 27696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34263 } }
I20260928 21:06:56.561048 27691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:56.562870 27697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34263 } } }
I20260928 21:06:56.562996 27697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:56.563458 27698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a6b6b76e75a4b9f8f7532c86997aae3. Latest consensus state: current_term: 1 leader_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6b6b76e75a4b9f8f7532c86997aae3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34263 } } }
I20260928 21:06:56.563519 27710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:56.563571 27698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:56.566349 27710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:56.572042 27710 catalog_manager.cc:1347] Generated new cluster ID: 41b3794c20664011b6cbcef40d861aa6
I20260928 21:06:56.572120 27710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:56.579384 27710 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:56.580619 27710 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:56.595678 27710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a6b6b76e75a4b9f8f7532c86997aae3: Generated new TSK 0
I20260928 21:06:56.596437 27710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:06:56.634924 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34263
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:56.747925 27717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:56.748246 27717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:56.748327 27717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:56.752595 27717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:56.752696 27717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:56.752794 27717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:56.758419 27717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:56.759574 27717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:56.760633 27717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:56.770859 27725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:56.770920 27722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:56.770887 27723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:56.771171 27717 server_base.cc:1034] running on GCE node
I20260928 21:06:56.771633 27717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:56.772302 27717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:56.773466 27717 hybrid_clock.cc:648] HybridClock initialized: now 1790629616773433 us; error 48 us; skew 500 ppm
Sep 28 21:06:56 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629616, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:56.776417 27717 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:06:56.777386 27717 webserver.cc:460] Webserver started at http://127.21.140.65:45255/ using document root <none> and password file <none>
I20260928 21:06:56.778035 27717 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:56.778105 27717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:56.778355 27717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:56.780449 27717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "e2ed2ca0de374a65abb16dc4f9d45b60"
format_stamp: "Formatted at 2026-09-28 21:06:56 on dist-test-slave-d5wc"
server_key: "bbeef1a77f562da4046b19152df35f10"
server_key_iv: "b56ad2e909936685f4746fe225e699ea"
server_key_version: "encryptionkey@0"
I20260928 21:06:56.781059 27717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "e2ed2ca0de374a65abb16dc4f9d45b60"
format_stamp: "Formatted at 2026-09-28 21:06:56 on dist-test-slave-d5wc"
server_key: "bbeef1a77f562da4046b19152df35f10"
server_key_iv: "b56ad2e909936685f4746fe225e699ea"
server_key_version: "encryptionkey@0"
I20260928 21:06:56.784902 27717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 21:06:56.787488 27732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:56.788803 27717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:06:56.788947 27717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "e2ed2ca0de374a65abb16dc4f9d45b60"
format_stamp: "Formatted at 2026-09-28 21:06:56 on dist-test-slave-d5wc"
server_key: "bbeef1a77f562da4046b19152df35f10"
server_key_iv: "b56ad2e909936685f4746fe225e699ea"
server_key_version: "encryptionkey@0"
I20260928 21:06:56.789072 27717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:56.818135 27717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:56.821439 27717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:56.821686 27717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:56.822382 27717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:56.823225 27717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:56.823307 27717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:56.823398 27717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:56.823450 27717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:56.832410 27717 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:37875
I20260928 21:06:56.832450 27846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:37875 every 8 connection(s)
I20260928 21:06:56.833300 27717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:56.842573 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27717
I20260928 21:06:56.842729 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:56.843020 22065 external_mini_cluster.cc:1428] Setting key 91c4db8d557c078e2e41333f07d9753a
Sep 28 21:06:56 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629616, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:56.848129 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:56.835287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:58909 (local address 127.21.140.126:34263)
0928 21:06:56.835433 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:56.835437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:56.836323 (+ 886us) server_negotiation.cc:407] Connection header received
0928 21:06:56.837628 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:56.837632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:56.837684 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:56.837765 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:56.839117 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:56.839673 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:56.840802 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:56.841068 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:56.841166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:56.843877 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:56.843881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:56.843884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:56.843917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:56.845775 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:56.846469 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:56.846473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:56.846475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:56.846569 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:56.846957 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:56.846966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:56.846968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:56.847152 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:56.847262 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:56.847804 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 21:06:56.847956 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:06:56.849005 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:56.835689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34263 (local address 127.21.140.65:58909)
0928 21:06:56.836156 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:56.836199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:56.837405 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:56.837932 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:56.837939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:56.838375 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:56.838970 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:56.838985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:56.839802 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:56.839804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:56.840679 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:56.840688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:56.841235 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:56.841238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:56.841384 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:56.842104 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:56.842121 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:56.843701 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:56.845932 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:56.845939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:56.845955 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:56.846338 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:56.846668 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:56.846673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:56.846675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:56.846842 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:56.847260 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:56.847267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:56.847544 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 21:06:56.847735 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 21:06:56.848077 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260928 21:06:56.850329 27847 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34263
I20260928 21:06:56.850584 27847 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:56.851115 27847 heartbeater.cc:511] Master 127.21.140.126:34263 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:06:56.852837 27655 ts_manager.cc:194] Registered new tserver with Master: e2ed2ca0de374a65abb16dc4f9d45b60 (127.21.140.65:37875)
I20260928 21:06:56.854226 27655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:58909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:06:56.873428 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34263
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:56.987679 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:56.987944 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:56.987990 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:56.992127 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:56.992183 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:56.992273 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:56.997676 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:56.998860 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:56.999891 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:57.010064 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:57.010133 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:57.010082 27852 server_base.cc:1034] running on GCE node
W20260928 21:06:57.010092 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:57.010677 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:57.011304 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:57.012504 27852 hybrid_clock.cc:648] HybridClock initialized: now 1790629617012490 us; error 27 us; skew 500 ppm
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.015676 27852 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:06:57.016675 27852 webserver.cc:460] Webserver started at http://127.21.140.66:42353/ using document root <none> and password file <none>
I20260928 21:06:57.017258 27852 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:57.017324 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:57.017510 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:57.019845 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "fe5d954ea6b54e89b0d8441cc0cd18f3"
format_stamp: "Formatted at 2026-09-28 21:06:57 on dist-test-slave-d5wc"
server_key: "774ce776022da36f1522dffa875f0a06"
server_key_iv: "5559303b055e8dcd52dbc47fa5ee100b"
server_key_version: "encryptionkey@0"
I20260928 21:06:57.020432 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "fe5d954ea6b54e89b0d8441cc0cd18f3"
format_stamp: "Formatted at 2026-09-28 21:06:57 on dist-test-slave-d5wc"
server_key: "774ce776022da36f1522dffa875f0a06"
server_key_iv: "5559303b055e8dcd52dbc47fa5ee100b"
server_key_version: "encryptionkey@0"
I20260928 21:06:57.024565 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 21:06:57.027369 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:57.028820 27852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:06:57.029045 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "fe5d954ea6b54e89b0d8441cc0cd18f3"
format_stamp: "Formatted at 2026-09-28 21:06:57 on dist-test-slave-d5wc"
server_key: "774ce776022da36f1522dffa875f0a06"
server_key_iv: "5559303b055e8dcd52dbc47fa5ee100b"
server_key_version: "encryptionkey@0"
I20260928 21:06:57.029165 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:57.067202 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:57.070596 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:57.070807 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:57.071403 27852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:57.072270 27852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:57.072319 27852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:57.072384 27852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:57.072417 27852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:57.081729 27852 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:35135
I20260928 21:06:57.081750 27981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:35135 every 8 connection(s)
I20260928 21:06:57.082747 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:57.091398 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27852
I20260928 21:06:57.091552 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:57.091861 22065 external_mini_cluster.cc:1428] Setting key 5d66cd5c280789453f08f5d0ad75202c
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:57.098354 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.084658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:49023 (local address 127.21.140.126:34263)
0928 21:06:57.084799 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.084803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.085634 (+ 831us) server_negotiation.cc:407] Connection header received
0928 21:06:57.087052 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.087055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.087106 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.087187 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.088404 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.088950 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.090135 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.090360 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.090451 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.093492 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:57.093504 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:57.093506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:57.093531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:57.095588 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.096344 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.096347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.096349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.096442 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.096979 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.096983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.096985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.097210 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:57.097349 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:57.098034 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 21:06:57.098175 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:06:57.099368 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.085005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34263 (local address 127.21.140.66:49023)
0928 21:06:57.085476 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:57.085518 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:57.086814 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:57.087365 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:57.087372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:57.087760 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:57.088247 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:57.088261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.089061 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.089063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:57.089944 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:57.089953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.090497 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.090500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:57.090662 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.091311 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:57.091328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:57.093224 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:57.095744 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:57.095754 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:57.095775 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:57.096219 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:57.096568 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:57.096573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:57.096576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:57.096796 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:57.097340 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:57.097348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:57.097701 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 21:06:57.097941 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 21:06:57.098351 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260928 21:06:57.100813 27982 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34263
I20260928 21:06:57.101104 27982 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:57.101667 27982 heartbeater.cc:511] Master 127.21.140.126:34263 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:57.102944 27655 ts_manager.cc:194] Registered new tserver with Master: fe5d954ea6b54e89b0d8441cc0cd18f3 (127.21.140.66:35135)
I20260928 21:06:57.103649 27655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:49023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:06:57.122723 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34263
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:57.271551 27987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:57.272002 27987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:57.272058 27987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:57.276858 27987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:57.276978 27987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:57.277051 27987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:57.283933 27987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:57.285495 27987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:57.287006 27987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:57.300300 27995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:57.300302 27992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:57.300300 27993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:57.300724 27987 server_base.cc:1034] running on GCE node
I20260928 21:06:57.301611 27987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:57.302644 27987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:57.303843 27987 hybrid_clock.cc:648] HybridClock initialized: now 1790629617303825 us; error 34 us; skew 500 ppm
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.307734 27987 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:06:57.309094 27987 webserver.cc:460] Webserver started at http://127.21.140.67:32785/ using document root <none> and password file <none>
I20260928 21:06:57.309904 27987 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:57.310057 27987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:57.310359 27987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:57.313313 27987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "ffc7044c6312487a87204de62b727300"
format_stamp: "Formatted at 2026-09-28 21:06:57 on dist-test-slave-d5wc"
server_key: "0af73c580480d0cae46e722533359c7f"
server_key_iv: "c4edd9a2d96bb753adec8b97f2ced11d"
server_key_version: "encryptionkey@0"
I20260928 21:06:57.314193 27987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "ffc7044c6312487a87204de62b727300"
format_stamp: "Formatted at 2026-09-28 21:06:57 on dist-test-slave-d5wc"
server_key: "0af73c580480d0cae46e722533359c7f"
server_key_iv: "c4edd9a2d96bb753adec8b97f2ced11d"
server_key_version: "encryptionkey@0"
I20260928 21:06:57.319752 27987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 21:06:57.324033 28002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:57.326040 27987 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:06:57.326289 27987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "ffc7044c6312487a87204de62b727300"
format_stamp: "Formatted at 2026-09-28 21:06:57 on dist-test-slave-d5wc"
server_key: "0af73c580480d0cae46e722533359c7f"
server_key_iv: "c4edd9a2d96bb753adec8b97f2ced11d"
server_key_version: "encryptionkey@0"
I20260928 21:06:57.326472 27987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:57.365078 27987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:57.369572 27987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:57.369848 27987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:57.370741 27987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:57.371866 27987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:57.371948 27987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:57.372030 27987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:57.372080 27987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:57.386459 27987 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:43845
I20260928 21:06:57.386478 28116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:43845 every 8 connection(s)
I20260928 21:06:57.387751 27987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:57.393323 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 27987
I20260928 21:06:57.393491 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:57.393874 22065 external_mini_cluster.cc:1428] Setting key 20dd16722eaafae0ce44580f191fb655
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:06:57.407932 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.390311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:51751 (local address 127.21.140.126:34263)
0928 21:06:57.390506 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.390511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.391814 (+ 1303us) server_negotiation.cc:407] Connection header received
0928 21:06:57.393956 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.393962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.394077 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.394167 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.396249 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.396931 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.398488 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.398732 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.398860 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.402280 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:57.402284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:57.402286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:57.402322 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:57.404492 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.405376 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.405385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.405386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.405488 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.405912 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.405916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.405918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.406416 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:57.406598 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:57.407505 (+ 907us) server_negotiation.cc:299] Negotiation successful
0928 21:06:57.407729 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 21:06:57.409478 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.390928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34263 (local address 127.21.140.67:51751)
0928 21:06:57.391572 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:57.391645 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:57.393580 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:57.394474 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:57.394484 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:57.395168 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:57.396035 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:57.396055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.397083 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.397087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:57.398338 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:57.398350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.398923 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.398927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:57.399155 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.399882 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:57.399903 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:57.401924 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:57.404749 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:57.404758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:57.404781 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:57.405207 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:57.405599 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:57.405603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:57.405605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:57.405797 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:57.406608 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:57.406622 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:57.407055 (+ 433us) client_negotiation.cc:769] Sending connection context
0928 21:06:57.407330 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 21:06:57.407987 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":165,"threads_started":1}
I20260928 21:06:57.411070 28117 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34263
I20260928 21:06:57.411417 28117 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:57.412175 28117 heartbeater.cc:511] Master 127.21.140.126:34263 requested a full tablet report, sending...
I20260928 21:06:57.413883 27655 ts_manager.cc:194] Registered new tserver with Master: ffc7044c6312487a87204de62b727300 (127.21.140.67:43845)
I20260928 21:06:57.414764 27655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:51751
I20260928 21:06:57.422333 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/test-admin.keytab.
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.444295 22065 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 21:06:57.448194 28124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:57.446285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.21.140.65:37875)
0928 21:06:57.446675 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.446683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.446714 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:06:57.446780 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.446787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.447050 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.447243 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.447942 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35720: BlockingRecv error: recv got EOF from 127.0.0.1:35720 (error 108)
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":146,"threads_started":1}
W20260928 21:06:57.448351 22065 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.449740 22065 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.451071 22065 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:57.464644 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.452500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.21.140.65:37875)
0928 21:06:57.452704 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.452709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.452729 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:06:57.452915 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.452924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.452983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.453084 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.453800 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.454770 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.455740 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.456277 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.456419 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.459756 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:57.459775 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:57.459796 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:57.459841 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:57.461909 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.462661 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.462668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.462669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.462773 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.463117 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.463121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.463122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.463647 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:57.463912 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:57.464128 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 21:06:57.464394 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 21:06:57.470515 28124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:57.469072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.21.140.65:37875)
0928 21:06:57.469285 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.469288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.469305 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:06:57.469510 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.469513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.469568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.469647 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.470345 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35738: BlockingRecv error: recv got EOF from 127.0.0.1:35738 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 21:06:57.471019 22065 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc)
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.472399 22065 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.473539 22065 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:57.484630 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.474864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.21.140.65:37875)
0928 21:06:57.475058 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.475062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.475080 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:06:57.475288 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.475292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.475351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.475438 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.476051 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.476715 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.477577 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.477882 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.478078 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.480853 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:57.480857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:57.480859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:57.480908 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:57.482436 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.482950 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.482955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.482957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.483053 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.483412 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.483415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.483416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.483659 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:57.483824 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:57.484147 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 21:06:57.484386 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 21:06:57.488281 28124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:06:57.487141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.21.140.65:37875)
0928 21:06:57.487377 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.487380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.487394 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:06:57.487502 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.487504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.487568 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.487642 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.488129 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35766: BlockingRecv error: recv got EOF from 127.0.0.1:35766 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260928 21:06:57.488502 22065 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629583469051-22065-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.489853 22065 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:57.491030 22065 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[27610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:06:57.503199 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:57.492434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.21.140.65:37875)
0928 21:06:57.492600 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:57.492604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:57.492621 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:06:57.492876 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:57.492881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:57.492946 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:57.493027 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:57.493857 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.494534 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.495538 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:57.495865 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:57.495993 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:57.499395 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:57.499400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:57.499403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:57.499448 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:57.501112 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.501695 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.501698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.501699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.501795 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:57.502180 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:57.502184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:57.502185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:57.502435 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:57.502585 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:57.502822 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 21:06:57.502951 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 21:06:57.504899 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27717
I20260928 21:06:57.512755 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27852
I20260928 21:06:57.520955 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27987
I20260928 21:06:57.528021 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 27623
2026-09-28T21:06:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2421 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[28137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[28137](info): set up 2 sockets
Sep 28 21:06:57 dist-test-slave-d5wc krb5kdc[28137](info): commencing operation
krb5kdc: starting...
W20260928 21:06:58.596660 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:06:58 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:06:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab.
Entry for principal oryx/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab.
I20260928 21:06:58.749660 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34327
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab
--principal=oryx/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:58.896521 28150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:58.897001 28150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:58.897094 28150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:58.901820 28150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:58.902029 28150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:58.902074 28150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:58.902114 28150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:58.902151 28150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:58.910624 28150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab
--principal=oryx/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:58.912292 28150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:58.913772 28150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:58.925582 28158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:58.925582 28156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:58.925614 28155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:58.925920 28150 server_base.cc:1034] running on GCE node
I20260928 21:06:58.926981 28150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:58.928802 28150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:58.930095 28150 hybrid_clock.cc:648] HybridClock initialized: now 1790629618930056 us; error 55 us; skew 500 ppm
Sep 28 21:06:58 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:58.934046 28150 init.cc:377] Logged in from keytab as oryx/127.21.140.126@KRBTEST.COM (short username oryx)
I20260928 21:06:58.935536 28150 webserver.cc:460] Webserver started at http://127.21.140.126:37505/ using document root <none> and password file <none>
I20260928 21:06:58.936380 28150 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:58.936506 28150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:58.936820 28150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:58.939836 28150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "e5c5bdfe48304bb3b161e60612c4a59c"
format_stamp: "Formatted at 2026-09-28 21:06:58 on dist-test-slave-d5wc"
server_key: "86e99334e08ba6d2c3c4b783e732312f"
server_key_iv: "99a2c32fd02983c2003f58e68376a139"
server_key_version: "encryptionkey@0"
I20260928 21:06:58.940665 28150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "e5c5bdfe48304bb3b161e60612c4a59c"
format_stamp: "Formatted at 2026-09-28 21:06:58 on dist-test-slave-d5wc"
server_key: "86e99334e08ba6d2c3c4b783e732312f"
server_key_iv: "99a2c32fd02983c2003f58e68376a139"
server_key_version: "encryptionkey@0"
I20260928 21:06:58.946552 28150 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 21:06:58.950349 28165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:58.952353 28150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:06:58.952597 28150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "e5c5bdfe48304bb3b161e60612c4a59c"
format_stamp: "Formatted at 2026-09-28 21:06:58 on dist-test-slave-d5wc"
server_key: "86e99334e08ba6d2c3c4b783e732312f"
server_key_iv: "99a2c32fd02983c2003f58e68376a139"
server_key_version: "encryptionkey@0"
I20260928 21:06:58.952796 28150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:58.972810 28150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:58.977381 28150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:58.977691 28150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:58.987509 28150 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:34327
I20260928 21:06:58.987879 28218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:34327 every 8 connection(s)
I20260928 21:06:58.989042 28150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:06:58.989959 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28150
I20260928 21:06:58.990219 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:06:58.990594 22065 external_mini_cluster.cc:1428] Setting key acc3b91ecaa18cf8e9ee9da9cd181b05
I20260928 21:06:58.993760 28219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 21:06:58 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:06:59.003309 28219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Bootstrap starting.
I20260928 21:06:59.007401 28219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Neither blocks nor log segments found. Creating new log.
I20260928 21:06:59.008713 28219 log.cc:824] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:06:59.010247 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:58.992549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.21.140.126:34327)
0928 21:06:58.993134 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:58.993160 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:58.993220 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:06:58.994356 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:58.994391 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:58.994987 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:58.995506 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:58.996366 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:58.998120 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:58.999164 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:58.999971 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.000176 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.003259 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:59.003267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:59.003288 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:59.003349 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:59.005889 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.006610 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.006619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.006627 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.006820 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.007149 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.007155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.007160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.007910 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:59.008165 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:59.008472 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:06:59.008887 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":169,"threads_started":1}
I20260928 21:06:59.011859 28219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: No bootstrap required, opened a new log
I20260928 21:06:59.017242 28219 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } }
I20260928 21:06:59.017800 28219 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:06:59.017913 28219 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c5bdfe48304bb3b161e60612c4a59c, State: Initialized, Role: FOLLOWER
I20260928 21:06:59.018620 28219 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } }
I20260928 21:06:59.018852 28219 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:06:59.018949 28219 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:06:59.019073 28219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:06:59.020416 28219 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } }
I20260928 21:06:59.020877 28219 leader_election.cc:302] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c5bdfe48304bb3b161e60612c4a59c; no voters:
I20260928 21:06:59.021327 28219 leader_election.cc:288] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:06:59.021550 28224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:06:59.021883 28224 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 LEADER]: Becoming Leader. State: Replica: e5c5bdfe48304bb3b161e60612c4a59c, State: Running, Role: LEADER
I20260928 21:06:59.022341 28224 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } }
I20260928 21:06:59.023252 28219 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:06:59.024578 28226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c5bdfe48304bb3b161e60612c4a59c. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } }
I20260928 21:06:59.024781 28225 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } }
I20260928 21:06:59.024950 28225 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:59.025281 28226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:06:59.025434 28233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:06:59.030293 28233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:06:59.040167 28233 catalog_manager.cc:1347] Generated new cluster ID: 0ca16b2057d44233b2d9ff3007c837fe
I20260928 21:06:59.040302 28233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:06:59.054564 28233 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:06:59.055893 28233 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:06:59.064719 28233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Generated new TSK 0
I20260928 21:06:59.066046 28233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab.
Entry for principal oryx/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab.
I20260928 21:06:59.110062 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34327
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:59.242523 28245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:59.242826 28245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:59.242871 28245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:59.247112 28245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:59.247182 28245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:59.247246 28245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:06:59.252698 28245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:59.253898 28245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:59.255091 28245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:59.267467 28251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.267584 28253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.267997 28250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:59.268059 28245 server_base.cc:1034] running on GCE node
I20260928 21:06:59.268611 28245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:59.269613 28245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:59.270840 28245 hybrid_clock.cc:648] HybridClock initialized: now 1790629619270781 us; error 62 us; skew 500 ppm
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:59.274309 28245 init.cc:377] Logged in from keytab as oryx/127.21.140.65@KRBTEST.COM (short username oryx)
I20260928 21:06:59.275367 28245 webserver.cc:460] Webserver started at http://127.21.140.65:41581/ using document root <none> and password file <none>
I20260928 21:06:59.276051 28245 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:59.276160 28245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:59.276423 28245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:59.278937 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "14a3755777754de48df817a03bf8a6d9"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "bed912c8f6fb1ddd6b3315bdd3e2b973"
server_key_iv: "f8e7c23361b6c3b2b5364afed802a154"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.279594 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "14a3755777754de48df817a03bf8a6d9"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "bed912c8f6fb1ddd6b3315bdd3e2b973"
server_key_iv: "f8e7c23361b6c3b2b5364afed802a154"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.284125 28245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:06:59.287122 28260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.288666 28245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:06:59.288823 28245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "14a3755777754de48df817a03bf8a6d9"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "bed912c8f6fb1ddd6b3315bdd3e2b973"
server_key_iv: "f8e7c23361b6c3b2b5364afed802a154"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.288969 28245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:59.301054 28245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:59.304606 28245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:59.304828 28245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:59.305466 28245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:59.306496 28245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:59.306571 28245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.306646 28245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:59.306699 28245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.315898 28245 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:40035
I20260928 21:06:59.315922 28374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:40035 every 8 connection(s)
I20260928 21:06:59.316855 28245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:06:59.319314 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28245
I20260928 21:06:59.319484 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:06:59.319825 22065 external_mini_cluster.cc:1428] Setting key 94f338e2dcd137f741193f97f9c89359
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.66@KRBTEST.COM" created.
I20260928 21:06:59.333679 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:59.319031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:35787 (local address 127.21.140.126:34327)
0928 21:06:59.319231 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:59.319236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:59.320269 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 21:06:59.321994 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:59.321998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:59.322056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:59.322141 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:59.323656 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.324392 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.326375 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.326639 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.326728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.329779 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:59.329783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:59.329784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:59.329815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:59.331412 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.332034 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.332040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.332042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.332130 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.332522 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.332525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.332527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.332727 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:59.332843 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:59.333354 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 21:06:59.333494 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 21:06:59.334640 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:59.319535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:35787)
0928 21:06:59.320095 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:59.320152 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:59.321730 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:59.322338 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:59.322346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:59.322739 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:59.323480 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:59.323503 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.324539 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.324546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:59.326245 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:59.326257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.326825 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.326832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:59.327098 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.327815 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:59.327843 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:59.329585 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:59.331605 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:59.331611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:59.331634 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:59.331911 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:59.332240 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:59.332243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:59.332245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:59.332408 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:59.332839 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:59.332845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:59.333108 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 21:06:59.333312 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 21:06:59.333668 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260928 21:06:59.336045 28375 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34327
I20260928 21:06:59.336314 28375 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:59.336941 28375 heartbeater.cc:511] Master 127.21.140.126:34327 requested a full tablet report, sending...
I20260928 21:06:59.338709 28183 ts_manager.cc:194] Registered new tserver with Master: 14a3755777754de48df817a03bf8a6d9 (127.21.140.65:40035)
I20260928 21:06:59.339920 28183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:35787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab.
Entry for principal oryx/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab.
I20260928 21:06:59.351419 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34327
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:59.471429 28380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:59.471747 28380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:59.471818 28380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:59.476068 28380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:59.476176 28380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:59.476267 28380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:06:59.482378 28380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:59.483567 28380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:59.484665 28380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:59.494980 28388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.495033 28385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.494992 28386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:59.495380 28380 server_base.cc:1034] running on GCE node
I20260928 21:06:59.495993 28380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:59.496667 28380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:59.497879 28380 hybrid_clock.cc:648] HybridClock initialized: now 1790629619497859 us; error 73 us; skew 500 ppm
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:59.500912 28380 init.cc:377] Logged in from keytab as oryx/127.21.140.66@KRBTEST.COM (short username oryx)
I20260928 21:06:59.501869 28380 webserver.cc:460] Webserver started at http://127.21.140.66:42693/ using document root <none> and password file <none>
I20260928 21:06:59.502517 28380 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:59.502585 28380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:59.502755 28380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:59.504842 28380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "aafc3b4ed0bb4412ac5d83d5724f6229"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "7dda77ea9683f3657b2544007161fbd6"
server_key_iv: "9dcf90e8d0dac6234df84e4e1f049187"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.505400 28380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "aafc3b4ed0bb4412ac5d83d5724f6229"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "7dda77ea9683f3657b2544007161fbd6"
server_key_iv: "9dcf90e8d0dac6234df84e4e1f049187"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.509416 28380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:06:59.512147 28395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.513497 28380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:06:59.513600 28380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "aafc3b4ed0bb4412ac5d83d5724f6229"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "7dda77ea9683f3657b2544007161fbd6"
server_key_iv: "9dcf90e8d0dac6234df84e4e1f049187"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.513698 28380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:59.529585 28380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:59.532863 28380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:59.533036 28380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:59.533561 28380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:59.534435 28380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:59.534495 28380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.534535 28380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:59.534551 28380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.543839 28380 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:32849
I20260928 21:06:59.543864 28509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:32849 every 8 connection(s)
I20260928 21:06:59.544853 28380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:06:59.548377 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28380
I20260928 21:06:59.548513 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:06:59.548892 22065 external_mini_cluster.cc:1428] Setting key 57f05dc0bca9d94f510f6e2a5b4bd1fc
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:06:59.560624 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:59.546953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:42943 (local address 127.21.140.126:34327)
0928 21:06:59.547116 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:59.547122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:59.548062 (+ 940us) server_negotiation.cc:407] Connection header received
0928 21:06:59.549763 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:59.549766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:59.549815 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:59.549902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:59.551555 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.552188 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.553370 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.553673 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.553793 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.556424 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:59.556430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:59.556433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:59.556464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:59.558313 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.558973 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.558979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.558983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.559081 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.559510 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.559513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.559514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.559713 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:59.559816 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:59.560363 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 21:06:59.560497 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 21:06:59.561522 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:59.547375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:42943)
0928 21:06:59.547850 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:59.547896 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:59.549573 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:59.550161 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:59.550169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:59.550643 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:59.551368 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:59.551392 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.552335 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.552339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:59.553233 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:59.553243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.553835 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.553839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:59.554091 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.554738 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:59.554756 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:59.556219 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:59.558492 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:59.558498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:59.558517 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:59.558865 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:59.559231 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:59.559234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:59.559235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:59.559396 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:59.559849 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:59.559855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:59.560117 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 21:06:59.560291 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 21:06:59.560563 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.67@KRBTEST.COM" created.
I20260928 21:06:59.562821 28510 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34327
I20260928 21:06:59.563078 28510 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:59.563570 28510 heartbeater.cc:511] Master 127.21.140.126:34327 requested a full tablet report, sending...
I20260928 21:06:59.564653 28183 ts_manager.cc:194] Registered new tserver with Master: aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849)
I20260928 21:06:59.565325 28183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:42943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab.
Entry for principal oryx/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab.
I20260928 21:06:59.582216 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34327
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:59.695089 28516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:59.695426 28516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:59.695500 28516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:59.699726 28516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:59.699820 28516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:59.699983 28516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:06:59.705425 28516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:34327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:06:59.706615 28516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:59.707643 28516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:59.717316 28524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.717340 28521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.717320 28522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:59.717470 28516 server_base.cc:1034] running on GCE node
I20260928 21:06:59.718132 28516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:59.718837 28516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:59.720022 28516 hybrid_clock.cc:648] HybridClock initialized: now 1790629619720005 us; error 29 us; skew 500 ppm
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:59.723131 28516 init.cc:377] Logged in from keytab as oryx/127.21.140.67@KRBTEST.COM (short username oryx)
I20260928 21:06:59.724150 28516 webserver.cc:460] Webserver started at http://127.21.140.67:43897/ using document root <none> and password file <none>
I20260928 21:06:59.724790 28516 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:59.724855 28516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:59.725073 28516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:59.727273 28516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "35838c8b76f145188b251f569c192617"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "deb6be6c1d556417676a126a45b991d2"
server_key_iv: "7e3cd21c5e38e944226b67e51d4209fd"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.727857 28516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "35838c8b76f145188b251f569c192617"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "deb6be6c1d556417676a126a45b991d2"
server_key_iv: "7e3cd21c5e38e944226b67e51d4209fd"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.731738 28516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 21:06:59.734318 28531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.735749 28516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:06:59.735903 28516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "35838c8b76f145188b251f569c192617"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "deb6be6c1d556417676a126a45b991d2"
server_key_iv: "7e3cd21c5e38e944226b67e51d4209fd"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.736078 28516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:06:59.759223 28516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:06:59.762581 28516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:06:59.762794 28516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:06:59.763401 28516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:06:59.764281 28516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:06:59.764351 28516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.764423 28516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:06:59.764452 28516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.773429 28516 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35435
I20260928 21:06:59.773452 28645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35435 every 8 connection(s)
I20260928 21:06:59.774410 28516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:06:59.779080 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28516
I20260928 21:06:59.779210 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:06:59.779498 22065 external_mini_cluster.cc:1428] Setting key f49c9446377f4e3d4d4038406f93bbf8
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:06:59.788743 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:59.776368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:59457 (local address 127.21.140.126:34327)
0928 21:06:59.776524 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:06:59.776527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:06:59.777335 (+ 808us) server_negotiation.cc:407] Connection header received
0928 21:06:59.778746 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:06:59.778750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:06:59.778831 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:06:59.778935 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:06:59.780299 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.781036 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.782234 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.782480 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.782598 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.785046 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:06:59.785049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:06:59.785052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:06:59.785087 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:06:59.786648 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.787171 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.787176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.787178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.787260 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:06:59.787613 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:06:59.787618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:06:59.787620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:06:59.787826 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:06:59.787933 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:06:59.788454 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 21:06:59.788578 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:06:59.789573 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:06:59.776745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:59457)
0928 21:06:59.777182 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 21:06:59.777224 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:06:59.778539 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:06:59.779112 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:06:59.779119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:06:59.779576 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:06:59.780162 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:59.780177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.781162 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.781171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:59.782115 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:06:59.782124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:06:59.782636 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:06:59.782640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:06:59.782816 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:06:59.783367 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:06:59.783383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:06:59.784864 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:06:59.786774 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:59.786780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:59.786795 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:59.787064 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:59.787354 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:06:59.787359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:06:59.787361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:06:59.787516 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:06:59.787952 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:06:59.787959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:06:59.788203 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 21:06:59.788393 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 21:06:59.788717 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260928 21:06:59.790800 28646 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34327
I20260928 21:06:59.791116 28646 heartbeater.cc:464] Registering TS with master...
I20260928 21:06:59.791607 28646 heartbeater.cc:511] Master 127.21.140.126:34327 requested a full tablet report, sending...
I20260928 21:06:59.792747 28182 ts_manager.cc:194] Registered new tserver with Master: 35838c8b76f145188b251f569c192617 (127.21.140.67:35435)
I20260928 21:06:59.793526 28182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:59457
I20260928 21:06:59.793731 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.140.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.125.keytab.
Entry for principal oryx/127.21.140.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.125.keytab.
I20260928 21:06:59.822513 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:45677
--webserver_interface=127.21.140.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.125.keytab
--principal=oryx/127.21.140.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:34327,127.21.140.125:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:06:59.937887 28651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:06:59.938246 28651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:06:59.938290 28651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:06:59.942528 28651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:06:59.942601 28651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:06:59.942621 28651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:06:59.942638 28651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:06:59.942655 28651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:06:59.948359 28651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:34327,127.21.140.125:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.125.keytab
--principal=oryx/127.21.140.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.140.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
W20260928 21:06:59.948589 28651 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.140.126:34327,127.21.140.125:45677'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 21:06:59.949447 28651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:06:59.950559 28651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:06:59.959322 28656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.959321 28659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:06:59.959317 28657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:06:59.959661 28651 server_base.cc:1034] running on GCE node
I20260928 21:06:59.960283 28651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:06:59.961404 28651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:06:59.962606 28651 hybrid_clock.cc:648] HybridClock initialized: now 1790629619962599 us; error 32 us; skew 500 ppm
Sep 28 21:06:59 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:06:59.965545 28651 init.cc:377] Logged in from keytab as oryx/127.21.140.125@KRBTEST.COM (short username oryx)
I20260928 21:06:59.966519 28651 webserver.cc:460] Webserver started at http://127.21.140.125:45037/ using document root <none> and password file <none>
I20260928 21:06:59.967065 28651 fs_manager.cc:359] Metadata directory not provided
I20260928 21:06:59.967127 28651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:06:59.967305 28651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:06:59.969321 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/instance:
uuid: "da816907c57947f0aab768c023fbe692"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "d10d4458452a764ae2111f1a2d4f8b91"
server_key_iv: "570764102a85b49230c0898bb370f7f8"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.969842 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal/instance:
uuid: "da816907c57947f0aab768c023fbe692"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "d10d4458452a764ae2111f1a2d4f8b91"
server_key_iv: "570764102a85b49230c0898bb370f7f8"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.973706 28651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:06:59.976238 28666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:06:59.977509 28651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:06:59.977636 28651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
uuid: "da816907c57947f0aab768c023fbe692"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "d10d4458452a764ae2111f1a2d4f8b91"
server_key_iv: "570764102a85b49230c0898bb370f7f8"
server_key_version: "encryptionkey@0"
I20260928 21:06:59.977727 28651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.003844 28651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:00.007372 28651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:00.007545 28651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:00.015404 28651 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.125:45677
I20260928 21:07:00.015727 28718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.125:45677 every 8 connection(s)
I20260928 21:07:00.016439 28651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
I20260928 21:07:00.019475 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28651
I20260928 21:07:00.019872 28719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:00.025555 28719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.125@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:00.036746 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.021197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.21.140.125:45677)
0928 21:07:00.021651 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.021668 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.021704 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:07:00.022465 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.022486 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.022839 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.023160 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.024248 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.025534 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.026740 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.027290 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.027479 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.031017 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.031023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.031035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.031068 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.033161 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.033807 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.033814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.033822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.034040 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.034345 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.034349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.034351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.034967 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.035227 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.035574 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.035908 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":161,"threads_started":1}
I20260928 21:07:00.038913 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.027746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.21.140.126:34327)
0928 21:07:00.027891 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.027896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.028325 (+ 429us) server_negotiation.cc:407] Connection header received
0928 21:07:00.028495 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.028498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.028560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.028663 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.029384 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.030287 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.031640 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.032086 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.032203 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.035168 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.035171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.035173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.035200 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.036891 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.037399 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.037403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.037405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.037510 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.037805 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.037809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.037811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.038140 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.038251 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.038608 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.038750 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:07:00.038930 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.027646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:49754)
0928 21:07:00.028187 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.028230 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.028387 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.028731 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.028739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.029116 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.029246 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.029252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.030477 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.030484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.031462 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.031476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.032245 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.032251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.032400 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.032973 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.032984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.034928 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.037034 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.037044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.037052 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.037281 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.037607 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.037611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.037613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.037707 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.038258 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.038267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.038388 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.038540 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.038738 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":95,"threads_started":1}
I20260928 21:07:00.040457 28719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629619, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.125@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
I20260928 21:07:00.049844 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.041135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.21.140.125:45677)
0928 21:07:00.041281 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.041284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.041296 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:07:00.041325 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.041327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.041359 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.041432 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.041870 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.042829 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.043634 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.043881 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.044021 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.046430 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.046434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.046436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.046485 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.048174 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.048653 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.048659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.048661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.048738 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.049019 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.049022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.049023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.049198 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.049307 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.049545 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.049707 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":55}
I20260928 21:07:00.049844 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.041026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55088)
0928 21:07:00.041135 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.041149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.041234 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.041484 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.041487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.041653 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.041751 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.041756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.042967 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.042970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.043483 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.043490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.044064 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.044067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.044147 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.044728 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.044740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.046262 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.048323 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.048328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.048330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.048549 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.048834 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.048838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.048840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.048920 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.049294 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.049297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.049377 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.049502 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.049652 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:07:00.053064 28719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Bootstrap starting.
I20260928 21:07:00.055969 28719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:00.056833 28719 log.cc:824] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:00.059100 28719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: No bootstrap required, opened a new log
I20260928 21:07:00.062394 28719 raft_consensus.cc:348] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } }
I20260928 21:07:00.062628 28719 raft_consensus.cc:374] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:00.062664 28719 raft_consensus.cc:729] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da816907c57947f0aab768c023fbe692, State: Initialized, Role: FOLLOWER
I20260928 21:07:00.063225 28719 consensus_queue.cc:260] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } }
I20260928 21:07:00.063899 28725 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } } }
I20260928 21:07:00.064067 28725 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:00.064568 28719 sys_catalog.cc:564] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:07:00.069664 28736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:07:00.069784 28736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:07:00.071936 28651 master_runner.cc:428] Detected that this master da816907c57947f0aab768c023fbe692 is joining an existing cluster
I20260928 21:07:00.071998 28651 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.140.125:45677
I20260928 21:07:00.082587 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.074114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:49760)
0928 21:07:00.074529 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.074545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.074729 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.075041 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.075044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.075312 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.075430 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.075435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.076194 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.076199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.076957 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.076967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.077440 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.077443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.077550 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.078133 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.078149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.078701 (+ 552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.080697 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.080700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.080702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.080999 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.081378 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.081380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.081381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.081505 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.082050 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.082053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.082142 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.082254 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.082393 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":113,"threads_started":1}
I20260928 21:07:00.082576 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.074210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.21.140.126:34327)
0928 21:07:00.074346 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.074349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.074640 (+ 291us) server_negotiation.cc:407] Connection header received
0928 21:07:00.074869 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.074873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.074928 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.075028 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.075550 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.076074 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.077081 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.077313 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.077429 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.078850 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.078856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.078861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.078889 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.080597 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.081125 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.081129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.081132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.081262 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.081622 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.081627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.081629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.081866 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.082007 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.082307 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.082427 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:07:00.086115 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.074634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.21.140.125:45677)
0928 21:07:00.074760 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.074764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.078115 (+ 3351us) server_negotiation.cc:407] Connection header received
0928 21:07:00.078267 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.078269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.078302 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.078381 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.078904 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.079678 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.080585 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.080974 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.081092 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.082409 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.082413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.082415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.082435 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.084358 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.084810 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.084814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.084817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.084906 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.085197 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.085200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.085201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.085397 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.085491 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.085811 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.085922 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":34}
I20260928 21:07:00.086568 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.074552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55096)
0928 21:07:00.077960 (+ 3408us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.077975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.078153 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.078393 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.078395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.078653 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.078734 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.078740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.079828 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.079833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.080449 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.080458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.081090 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.081094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.081224 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.081722 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.081736 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.082279 (+ 543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.084469 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.084472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.084475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.084708 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.085018 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.085020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.085021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.085088 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.085552 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.085554 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.085640 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.086291 (+ 651us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.086413 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3316,"mutex_wait_us":5,"thread_start_us":71,"threads_started":1}
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.126@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
I20260928 21:07:00.097190 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.087601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.21.140.125:45677)
0928 21:07:00.087722 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.087726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.088203 (+ 477us) server_negotiation.cc:407] Connection header received
0928 21:07:00.088371 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.088373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.088413 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.088478 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.089242 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.089820 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.090734 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.090961 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.091069 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.093798 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.093802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.093805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.093827 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.095346 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.095856 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.095860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.095862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.095943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.096230 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.096235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.096237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.096443 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.096553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.096903 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.097016 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:07:00.097209 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.087572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55102)
0928 21:07:00.088052 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.088111 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.088244 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.088540 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.088547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.088982 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.089135 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.089143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.089925 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.089930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.090612 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.090622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.091072 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.091076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.091202 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.091776 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.091796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.093654 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.095464 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.095475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.095482 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.095751 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.096043 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.096045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.096047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.096139 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.096554 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.096561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.096696 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.096843 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.097054 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":102,"threads_started":1}
I20260928 21:07:00.099110 28182 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.140.125:45677: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "da816907c57947f0aab768c023fbe692"
member_type: NON_VOTER
last_known_addr {
host: "127.21.140.125"
port: 45677
}
attrs {
promote: true
}
}
dest_uuid: "e5c5bdfe48304bb3b161e60612c4a59c"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.21.140.124@KRBTEST.COM; defaulting to no policy
I20260928 21:07:00.099794 28182 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } }
I20260928 21:07:00.101787 28748 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER da816907c57947f0aab768c023fbe692 (127.21.140.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.124@KRBTEST.COM" created.
I20260928 21:07:00.102533 28693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:00.103626 28748 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.140.125:45677
I20260928 21:07:00.104070 28651 master.cc:537] Master@127.21.140.125:45677 shutting down...
I20260928 21:07:00.104300 28693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Refusing update from remote peer e5c5bdfe48304bb3b161e60612c4a59c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 21:07:00.105145 28749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } }
I20260928 21:07:00.105273 28749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:00.105779 28748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 21:07:00.108282 28651 raft_consensus.cc:2229] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:00.108569 28651 raft_consensus.cc:2258] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:00.108675 28651 tablet_replica.cc:330] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: stopping tablet replica
W20260928 21:07:00.108690 28169 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 21:07:00.109722 28169 consensus_peers.cc:487] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c -> Peer da816907c57947f0aab768c023fbe692 (127.21.140.125:45677): Couldn't send request to peer da816907c57947f0aab768c023fbe692. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:07:00.123896 28651 master.cc:552] Master@127.21.140.125:45677 shutdown complete.
I20260928 21:07:00.124054 28651 master_runner.cc:305] Clearing existing system tablet
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.124.keytab.
Entry for principal oryx/127.21.140.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.124.keytab.
I20260928 21:07:00.124377 28651 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.124459 28651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.125224 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:34455
--webserver_interface=127.21.140.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.124.keytab
--principal=oryx/127.21.140.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:07:00.125869 28651 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:00.126952 28756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.127180 28651 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.127348 28651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:00.127409 28651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
uuid: "da816907c57947f0aab768c023fbe692"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "d10d4458452a764ae2111f1a2d4f8b91"
server_key_iv: "570764102a85b49230c0898bb370f7f8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.129868 28651 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:00.132108 28651 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:07:00.132234 28651 log.cc:1196] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 21:07:00.132486 28651 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Deleting consensus metadata
I20260928 21:07:00.132684 28651 master_runner.cc:315] Copying system tablet from 127.21.140.126:34327
I20260928 21:07:00.133872 28651 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: tablet copy: Beginning tablet copy session from remote peer at address 127.21.140.126:34327
I20260928 21:07:00.142807 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.134615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.21.140.126:34327)
0928 21:07:00.134748 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.134753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.134992 (+ 239us) server_negotiation.cc:407] Connection header received
0928 21:07:00.135180 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.135182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.135239 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.135332 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.135772 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.136298 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.137209 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.137448 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.137570 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.139225 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.139229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.139232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.139258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.141036 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.141473 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.141476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.141478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.141570 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.141861 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.141865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.141866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.142146 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.142252 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.142521 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.142642 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:07:00.142822 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.134522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:49774)
0928 21:07:00.134893 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.134906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.135059 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.135348 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.135350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.135563 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.135654 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.135660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.136410 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.136417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.137013 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.137021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.137568 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.137572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.137665 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.138687 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.138700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.139075 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.141153 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.141158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.141161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.141369 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.141665 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.141670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.141672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.141761 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.142252 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.142255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.142356 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.142464 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.142620 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":113,"threads_started":1}
I20260928 21:07:00.143752 28203 tablet_copy_service.cc:140] P e5c5bdfe48304bb3b161e60612c4a59c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer da816907c57947f0aab768c023fbe692 ({username='oryx', principal='oryx/127.21.140.125@KRBTEST.COM'} at 127.0.0.1:49774)
I20260928 21:07:00.143935 28203 tablet_copy_service.cc:161] P e5c5bdfe48304bb3b161e60612c4a59c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer da816907c57947f0aab768c023fbe692 at {username='oryx', principal='oryx/127.21.140.125@KRBTEST.COM'} at 127.0.0.1:49774: session id = da816907c57947f0aab768c023fbe692-00000000000000000000000000000000
I20260928 21:07:00.145346 28203 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:07:00.146876 28651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:07:00.148968 28651 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: tablet copy: Starting download of 0 data blocks...
I20260928 21:07:00.149194 28651 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: tablet copy: Starting download of 1 WAL segments...
I20260928 21:07:00.150965 28651 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:07:00.152573 28203 tablet_copy_service.cc:342] P e5c5bdfe48304bb3b161e60612c4a59c: Request end of tablet copy session da816907c57947f0aab768c023fbe692-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.140.125@KRBTEST.COM'} at 127.0.0.1:49774
I20260928 21:07:00.152704 28203 tablet_copy_service.cc:433] P e5c5bdfe48304bb3b161e60612c4a59c: ending tablet copy session da816907c57947f0aab768c023fbe692-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer da816907c57947f0aab768c023fbe692
I20260928 21:07:00.159940 28651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:00.161775 28766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.161949 28651 server_base.cc:1034] running on GCE node
W20260928 21:07:00.161775 28763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.166255 28764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.166574 28651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:00.167102 28651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:00.168282 28651 hybrid_clock.cc:648] HybridClock initialized: now 1790629620168263 us; error 44 us; skew 500 ppm
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:00.169821 28651 init.cc:377] Logged in from keytab as oryx/127.21.140.125@KRBTEST.COM (short username oryx)
I20260928 21:07:00.170369 28651 webserver.cc:460] Webserver started at http://127.21.140.125:34607/ using document root <none> and password file <none>
I20260928 21:07:00.170585 28651 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.170640 28651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.171851 28651 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:07:00.172853 28773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.173233 28651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:00.173331 28651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
uuid: "da816907c57947f0aab768c023fbe692"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "d10d4458452a764ae2111f1a2d4f8b91"
server_key_iv: "570764102a85b49230c0898bb370f7f8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.173437 28651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.188493 28651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:00.191371 28651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:00.196746 28651 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.125:45677
I20260928 21:07:00.197360 28651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
I20260928 21:07:00.197749 28825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.125:45677 every 8 connection(s)
I20260928 21:07:00.199002 28826 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:07:00.199833 28826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Bootstrap starting.
I20260928 21:07:00.208789 28826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:07:00.209141 28826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Bootstrap complete.
I20260928 21:07:00.209671 28826 raft_consensus.cc:348] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } }
I20260928 21:07:00.209820 28826 raft_consensus.cc:729] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da816907c57947f0aab768c023fbe692, State: Initialized, Role: LEARNER
I20260928 21:07:00.209945 28826 consensus_queue.cc:260] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } }
I20260928 21:07:00.210377 28829 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } }
I20260928 21:07:00.210565 28829 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:07:00.210564 28826 sys_catalog.cc:564] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:00.214078 28840 catalog_manager.cc:1259] Loaded cluster ID: 0ca16b2057d44233b2d9ff3007c837fe
I20260928 21:07:00.214169 28840 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: loading cluster ID for follower catalog manager: success
I20260928 21:07:00.215839 28840 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: acquiring CA information for follower catalog manager: success
I20260928 21:07:00.217213 28840 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:07:00.287849 28755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:00.288130 28755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:00.288174 28755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:00.292649 28755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:00.292750 28755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:00.292779 28755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:00.292805 28755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:00.292852 28755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:00.298666 28755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.124.keytab
--principal=oryx/127.21.140.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.140.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:00.299817 28755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:00.300868 28755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:00.309726 28845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.309716 28846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.309754 28755 server_base.cc:1034] running on GCE node
W20260928 21:07:00.309716 28848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.310698 28755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:00.311865 28755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:00.313042 28755 hybrid_clock.cc:648] HybridClock initialized: now 1790629620313012 us; error 49 us; skew 500 ppm
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:00.316198 28755 init.cc:377] Logged in from keytab as oryx/127.21.140.124@KRBTEST.COM (short username oryx)
I20260928 21:07:00.317281 28755 webserver.cc:460] Webserver started at http://127.21.140.124:38497/ using document root <none> and password file <none>
I20260928 21:07:00.317901 28755 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.318032 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.318271 28755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:00.320389 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/instance:
uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
format_stamp: "Formatted at 2026-09-28 21:07:00 on dist-test-slave-d5wc"
server_key: "f3c71517bcab7955dc06cabc6c42a8da"
server_key_iv: "50c792c924055e964aef3e8c3a20fdb8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.321004 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal/instance:
uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
format_stamp: "Formatted at 2026-09-28 21:07:00 on dist-test-slave-d5wc"
server_key: "f3c71517bcab7955dc06cabc6c42a8da"
server_key_iv: "50c792c924055e964aef3e8c3a20fdb8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.325457 28755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 21:07:00.328058 28855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.329339 28755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:07:00.329452 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
format_stamp: "Formatted at 2026-09-28 21:07:00 on dist-test-slave-d5wc"
server_key: "f3c71517bcab7955dc06cabc6c42a8da"
server_key_iv: "50c792c924055e964aef3e8c3a20fdb8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.329597 28755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.341521 28755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:00.342742 28375 heartbeater.cc:503] Master 127.21.140.126:34327 was elected leader, sending a full tablet report...
I20260928 21:07:00.345130 28755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:00.345335 28755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:00.352278 28755 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.124:34455
I20260928 21:07:00.352437 28908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.124:34455 every 8 connection(s)
I20260928 21:07:00.353474 28755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
I20260928 21:07:00.353679 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28755
I20260928 21:07:00.356817 28909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:00.361452 28909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.140.124@KRBTEST.COM
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.124@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:00.369822 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.355244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.21.140.124:34455)
0928 21:07:00.355693 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.355711 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.355753 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:07:00.356610 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.356632 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.357057 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.357440 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.358375 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.359481 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.360374 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.360871 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.361069 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.364231 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.364237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.364251 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.364285 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.366192 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.366731 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.366736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.366741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.366865 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.367313 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.367319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.367326 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.367966 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.368234 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.368581 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.368910 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260928 21:07:00.373373 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.363726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.21.140.126:34327)
0928 21:07:00.363865 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.363869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.364205 (+ 336us) server_negotiation.cc:407] Connection header received
0928 21:07:00.364375 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.364379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.364435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.364532 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.365169 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.365698 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.366880 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.367126 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.367242 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.369650 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.369654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.369657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.369692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.371436 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.371948 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.371952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.371954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.372044 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.372399 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.372403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.372404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.372613 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.372727 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.373090 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.373207 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:07:00.373394 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.363582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:49786)
0928 21:07:00.364056 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.364115 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.364273 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.364562 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.364568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.364917 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.365045 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.365050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.365812 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.365817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.366764 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.366773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.367245 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.367248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.367361 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.367952 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.367966 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.369503 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.371567 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.371574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.371579 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.371805 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.372167 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.372190 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.372195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.372298 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.372723 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.372729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.372845 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.372968 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.373197 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":142,"threads_started":1}
I20260928 21:07:00.374591 28909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.124@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
I20260928 21:07:00.384019 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.375110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55104)
0928 21:07:00.375551 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.375564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.375676 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.376011 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.376013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.376176 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.376275 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.376280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.377232 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.377235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.377838 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.377847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.378448 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.378452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.378531 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.379082 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.379095 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.380594 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.382454 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.382457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.382460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.382669 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.382977 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.382980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.382982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.383068 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.383465 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.383468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.383570 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.383684 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.383842 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377}
I20260928 21:07:00.384022 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.375340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.21.140.125:45677)
0928 21:07:00.375596 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.375609 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.375622 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:07:00.375802 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.375805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.375847 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.375916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.376382 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.377097 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.378039 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.378316 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.378447 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.380759 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.380762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.380765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.380789 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.382328 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.382778 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.382782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.382784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.382863 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.383168 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.383171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.383173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.383362 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.383474 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.383737 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.383859 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260928 21:07:00.384850 28909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.124@KRBTEST.COM for oryx/127.21.140.124@KRBTEST.COM
I20260928 21:07:00.393568 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.385428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49192)
0928 21:07:00.385546 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.385558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.385619 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.385850 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.385851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.386061 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.386147 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.386152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.387409 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.387411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.387925 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.387934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.388408 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.388411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.388539 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.389040 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.389052 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.390302 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.392053 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.392056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.392057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.392324 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.392582 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.392584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.392585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.392695 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.393126 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.393128 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.393205 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.393328 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.393437 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":42}
I20260928 21:07:00.393661 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.385545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.21.140.124:34455)
0928 21:07:00.385670 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.385674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.385686 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:07:00.385717 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.385719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.385755 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.385828 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.386249 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.387288 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.388069 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.388302 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.388411 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.390456 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.390459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.390462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.390484 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.391887 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.392417 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.392419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.392421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.392496 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.392785 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.392787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.392788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.393016 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.393102 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.393412 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.393519 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260928 21:07:00.396749 28909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Bootstrap starting.
I20260928 21:07:00.399176 28909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:00.399942 28909 log.cc:824] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:00.402226 28909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: No bootstrap required, opened a new log
I20260928 21:07:00.405493 28909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } }
I20260928 21:07:00.405759 28909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:00.405797 28909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b04c6aeb3c4c2183e04ae82b8ef6ef, State: Initialized, Role: FOLLOWER
I20260928 21:07:00.406404 28909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } }
I20260928 21:07:00.407084 28915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } } }
I20260928 21:07:00.407224 28915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:00.407832 28909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:07:00.413414 28926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:07:00.413542 28926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:07:00.417062 28755 master_runner.cc:428] Detected that this master 20b04c6aeb3c4c2183e04ae82b8ef6ef is joining an existing cluster
I20260928 21:07:00.417167 28755 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.140.124:34455
I20260928 21:07:00.423708 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28245
I20260928 21:07:00.430445 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.419925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.21.140.126:34327)
0928 21:07:00.420080 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.420085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.420327 (+ 242us) server_negotiation.cc:407] Connection header received
0928 21:07:00.420480 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.420483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.420545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.420632 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.421345 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.422227 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.423592 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.423934 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.424028 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.425724 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.425728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.425731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.425759 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.427955 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.428787 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.428794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.428798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.428896 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.429245 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.429252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.429256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.429523 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.429601 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.430039 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.430188 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:07:00.431165 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.420708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.21.140.125:45677)
0928 21:07:00.420831 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.420835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.421066 (+ 231us) server_negotiation.cc:407] Connection header received
0928 21:07:00.421233 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.421236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.421293 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.421344 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.421910 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.422705 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.424932 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.425295 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.425398 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.426982 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.426986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.426989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.427018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.429330 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.429822 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.429826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.429828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.429908 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.430230 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.430234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.430235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.430437 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.430516 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.430882 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.431016 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:07:00.431735 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.420398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55116)
0928 21:07:00.420991 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.421005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.421121 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.421484 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.421487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.421680 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.421781 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.421788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.422921 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.422924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.423718 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.423729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.425514 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.425518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.425760 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.426327 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.426344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.426829 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.429454 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.429457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.429460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.429707 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.430043 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.430046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.430048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.430130 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.430595 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.430597 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.430697 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.431339 (+ 642us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.431483 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260928 21:07:00.432300 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.419851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:49794)
0928 21:07:00.420241 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.420257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.420369 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.420764 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.420768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.421032 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.421194 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.421202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.422373 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.422376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.423431 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.423441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.424218 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.424221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.424308 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.424900 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.424916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.425552 (+ 636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.428090 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.428096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.428101 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.428397 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.429026 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.429032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.429037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.429142 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.429677 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.429682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.429797 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.430934 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.431119 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":42,"spinlock_wait_cycles":3456,"thread_start_us":216,"threads_started":1}
I20260928 21:07:00.432695 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:40035
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:34327
--webserver_port=41581
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629618, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.126@KRBTEST.COM for oryx/127.21.140.124@KRBTEST.COM
I20260928 21:07:00.451321 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.419859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49204)
0928 21:07:00.420605 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.420620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.420729 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.421069 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.421072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.421312 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.421476 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.421483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.422465 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.422468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.430788 (+ 8320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.430799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.431476 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.431479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.431571 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.432055 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.432071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.432449 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.434434 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.434437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.434439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.434655 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.450257 (+ 15602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.450260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.450263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.450358 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.450793 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.450796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.450869 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.450959 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.451114 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":6301,"thread_start_us":153,"threads_started":1}
I20260928 21:07:00.451786 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.419914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.21.140.124:34455)
0928 21:07:00.420024 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.420030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.420678 (+ 648us) server_negotiation.cc:407] Connection header received
0928 21:07:00.420819 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.420822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.420864 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.420930 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.421583 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.422342 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.430906 (+ 8564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.431255 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.431339 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.432587 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.432592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.432594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.432616 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.434327 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.434760 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.434765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.434768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.450128 (+ 15360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.450443 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.450446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.450448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.450672 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.450753 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.451551 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.451661 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:07:00.458097 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.437792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49210)
0928 21:07:00.438095 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.438112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.438205 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.439220 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.439224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.439453 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.439577 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.439584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.441106 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.441110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.442362 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.442374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.443011 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.443015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.443321 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.443874 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.443897 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.445766 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.456170 (+ 10404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.456175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.456180 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.456499 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.456846 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.456849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.456852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.456964 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.457450 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.457455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.457577 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.457661 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.457826 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260928 21:07:00.458781 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.438072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.21.140.124:34455)
0928 21:07:00.438615 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.438623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.438637 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:00.438706 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.438709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.438779 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.439728 (+ 949us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.439838 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.440971 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.442514 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.442869 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.443572 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.446199 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.446203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.446206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.446233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.454976 (+ 8743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.456621 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.456625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.456627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.456743 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.457085 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.457089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.457091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.457327 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.458396 (+ 1069us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.458470 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.458609 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":103,"threads_started":1}
I20260928 21:07:00.459743 28182 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.140.124:34455: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
member_type: NON_VOTER
last_known_addr {
host: "127.21.140.124"
port: 34455
}
attrs {
promote: true
}
}
dest_uuid: "e5c5bdfe48304bb3b161e60612c4a59c"
cas_config_opid_index: 5
I20260928 21:07:00.460383 28182 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } }
I20260928 21:07:00.461454 28748 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.462491 28883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:00.462894 28748 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.140.124:34455
I20260928 21:07:00.463068 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.463157 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:00.463966 28883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Refusing update from remote peer e5c5bdfe48304bb3b161e60612c4a59c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:07:00.464180 28755 master.cc:537] Master@127.21.140.124:34455 shutting down...
I20260928 21:07:00.465773 28751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:07:00.467813 28170 consensus_peers.cc:487] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c -> Peer 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124:34455): Couldn't send request to peer 20b04c6aeb3c4c2183e04ae82b8ef6ef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:07:00.471776 28755 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:00.472096 28755 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:00.472204 28755 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: stopping tablet replica
I20260928 21:07:00.473266 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.461631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55118)
0928 21:07:00.462112 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.462131 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.462221 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.465100 (+ 2879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.465103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.465123 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:00.465216 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.465223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.466424 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.466427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.467777 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.467790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.472733 (+ 4943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.472737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.472864 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.472932 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.473028 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.473126 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400}
I20260928 21:07:00.473309 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.463439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.21.140.125:45677)
0928 21:07:00.463555 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.463560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.463573 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:07:00.463625 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.463628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.463686 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.463787 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:00.465357 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.466193 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.468024 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.468706 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.468802 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.468976 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.473080 (+ 4104us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.473181 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:07:00.473850 28793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 LEARNER]: Refusing update from remote peer e5c5bdfe48304bb3b161e60612c4a59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:07:00.474491 28751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 21:07:00.476572 28793 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.478749 28938 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.478960 28938 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:07:00.479296 28829 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c5bdfe48304bb3b161e60612c4a59c. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: NON_VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: true } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.479420 28829 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:07:00.479918 28751 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: attempting to promote NON_VOTER da816907c57947f0aab768c023fbe692 to VOTER
I20260928 21:07:00.480404 28751 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } }
I20260928 21:07:00.482359 28793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 LEARNER]: Refusing update from remote peer e5c5bdfe48304bb3b161e60612c4a59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 21:07:00.482506 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.481939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49216)
0928 21:07:00.482249 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.482336 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":142,"threads_started":1}
I20260928 21:07:00.482741 28749 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 21:07:00.484613 28749 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da816907c57947f0aab768c023fbe692 (127.21.140.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.486294 28749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.486414 28749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:00.486532 28749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.486608 28749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:00.487210 28794 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, da816907c57947f0aab768c023fbe692 (127.21.140.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.489058 28829 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.489182 28829 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:00.489992 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.489669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49224)
0928 21:07:00.489805 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.489876 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 21:07:00.490186 28170 consensus_peers.cc:487] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c -> Peer 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124:34455): Couldn't send request to peer 20b04c6aeb3c4c2183e04ae82b8ef6ef. Status: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:07:00.502116 28755 master.cc:552] Master@127.21.140.124:34455 shutdown complete.
I20260928 21:07:00.502188 28755 master_runner.cc:305] Clearing existing system tablet
I20260928 21:07:00.502430 28755 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.502488 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.503849 28755 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:07:00.504748 28941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.504979 28755 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.505229 28755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:00.505296 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
format_stamp: "Formatted at 2026-09-28 21:07:00 on dist-test-slave-d5wc"
server_key: "f3c71517bcab7955dc06cabc6c42a8da"
server_key_iv: "50c792c924055e964aef3e8c3a20fdb8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.507408 28755 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:00.510116 28755 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:07:00.510215 28755 log.cc:1196] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 21:07:00.510514 28755 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Deleting consensus metadata
I20260928 21:07:00.510676 28755 master_runner.cc:315] Copying system tablet from 127.21.140.126:34327
I20260928 21:07:00.511935 28755 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: tablet copy: Beginning tablet copy session from remote peer at address 127.21.140.126:34327
I20260928 21:07:00.523092 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.512512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:49796)
0928 21:07:00.514179 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.514197 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.514336 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.514778 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.514781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.515017 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.515166 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.515173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.516259 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.516262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.517194 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.517204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.517945 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.517949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.518098 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.518697 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.518712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.519165 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.521183 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.521187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.521189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.521459 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.521782 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.521785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.521787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.521876 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.522458 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.522461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.522570 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.522681 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.522833 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"thread_start_us":92,"threads_started":1}
I20260928 21:07:00.523136 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.512628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.21.140.126:34327)
0928 21:07:00.512758 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.512763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.514317 (+ 1554us) server_negotiation.cc:407] Connection header received
0928 21:07:00.514488 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.514492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.514561 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.514620 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.515306 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.516112 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.517333 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.517658 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.517733 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.519335 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.519339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.519343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.519372 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.521079 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.521580 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.521583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.521585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.521676 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.522008 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.522011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.522012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.522338 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.522457 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.522766 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.522949 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:07:00.524014 28203 tablet_copy_service.cc:140] P e5c5bdfe48304bb3b161e60612c4a59c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20b04c6aeb3c4c2183e04ae82b8ef6ef ({username='oryx', principal='oryx/127.21.140.124@KRBTEST.COM'} at 127.0.0.1:49796)
I20260928 21:07:00.524122 28203 tablet_copy_service.cc:161] P e5c5bdfe48304bb3b161e60612c4a59c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20b04c6aeb3c4c2183e04ae82b8ef6ef at {username='oryx', principal='oryx/127.21.140.124@KRBTEST.COM'} at 127.0.0.1:49796: session id = 20b04c6aeb3c4c2183e04ae82b8ef6ef-00000000000000000000000000000000
I20260928 21:07:00.525219 28203 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:07:00.526702 28755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:07:00.529253 28755 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: tablet copy: Starting download of 0 data blocks...
I20260928 21:07:00.529456 28755 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: tablet copy: Starting download of 1 WAL segments...
I20260928 21:07:00.531179 28755 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:07:00.532967 28203 tablet_copy_service.cc:342] P e5c5bdfe48304bb3b161e60612c4a59c: Request end of tablet copy session 20b04c6aeb3c4c2183e04ae82b8ef6ef-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.140.124@KRBTEST.COM'} at 127.0.0.1:49796
I20260928 21:07:00.533072 28203 tablet_copy_service.cc:433] P e5c5bdfe48304bb3b161e60612c4a59c: ending tablet copy session 20b04c6aeb3c4c2183e04ae82b8ef6ef-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20b04c6aeb3c4c2183e04ae82b8ef6ef
I20260928 21:07:00.540608 28755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:00.542660 28948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.547260 28949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.547437 28951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.549383 28755 server_base.cc:1034] running on GCE node
I20260928 21:07:00.549607 28755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:00.549927 28755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:00.551131 28755 hybrid_clock.cc:648] HybridClock initialized: now 1790629620551104 us; error 42 us; skew 500 ppm
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:00.553117 28755 init.cc:377] Logged in from keytab as oryx/127.21.140.124@KRBTEST.COM (short username oryx)
I20260928 21:07:00.553596 28755 webserver.cc:460] Webserver started at http://127.21.140.124:36207/ using document root <none> and password file <none>
I20260928 21:07:00.553791 28755 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.553836 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.555269 28755 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:00.556391 28958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.556638 28755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:00.556740 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
format_stamp: "Formatted at 2026-09-28 21:07:00 on dist-test-slave-d5wc"
server_key: "f3c71517bcab7955dc06cabc6c42a8da"
server_key_iv: "50c792c924055e964aef3e8c3a20fdb8"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.556824 28755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.567516 28510 heartbeater.cc:503] Master 127.21.140.126:34327 was elected leader, sending a full tablet report...
I20260928 21:07:00.575048 28755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:00.577212 28755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:00.582297 28755 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.124:34455
I20260928 21:07:00.582461 29011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.124:34455 every 8 connection(s)
I20260928 21:07:00.582880 28755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
I20260928 21:07:00.584617 29012 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:07:00.585685 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Bootstrap starting.
W20260928 21:07:00.593154 28935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:00.593468 28935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:00.593559 28935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:07:00.597139 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:07:00.597452 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: Bootstrap complete.
W20260928 21:07:00.597848 28935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:00.597927 28935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:00.598053 28935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:00.598096 29012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } }
I20260928 21:07:00.598255 29012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20b04c6aeb3c4c2183e04ae82b8ef6ef, State: Initialized, Role: LEARNER
I20260928 21:07:00.598376 29012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } }
I20260928 21:07:00.598768 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:00.598889 29012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:00.598979 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: This master's current role is: LEARNER
I20260928 21:07:00.601951 29029 catalog_manager.cc:1259] Loaded cluster ID: 0ca16b2057d44233b2d9ff3007c837fe
I20260928 21:07:00.602102 29029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: loading cluster ID for follower catalog manager: success
I20260928 21:07:00.604213 29029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: acquiring CA information for follower catalog manager: success
I20260928 21:07:00.605604 29029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:07:00.606024 28935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:40035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=41581
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:00.607072 28935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:00.608098 28935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:00.618359 29034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.618571 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.618414 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.618528 28935 server_base.cc:1034] running on GCE node
I20260928 21:07:00.619094 28935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:00.619836 28935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:00.621024 28935 hybrid_clock.cc:648] HybridClock initialized: now 1790629620620995 us; error 60 us; skew 500 ppm
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:00.624267 28935 init.cc:377] Logged in from keytab as oryx/127.21.140.65@KRBTEST.COM (short username oryx)
I20260928 21:07:00.625388 28935 webserver.cc:460] Webserver started at http://127.21.140.65:41581/ using document root <none> and password file <none>
I20260928 21:07:00.626107 28935 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.626207 28935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.630769 28935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:00.632862 29041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.634162 28935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:07:00.634258 28935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "14a3755777754de48df817a03bf8a6d9"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "bed912c8f6fb1ddd6b3315bdd3e2b973"
server_key_iv: "f8e7c23361b6c3b2b5364afed802a154"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.634775 28935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.650928 28935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:00.653841 28935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:00.654060 28935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:00.654599 28935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:00.655388 28935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:00.655434 28935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.655473 28935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:00.655488 28935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.664351 28935 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:40035
I20260928 21:07:00.664405 29154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:40035 every 8 connection(s)
I20260928 21:07:00.665367 28935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:00.672254 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 28935
I20260928 21:07:00.672449 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28380
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.124@KRBTEST.COM
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
I20260928 21:07:00.682961 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:32849
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:34327
--webserver_port=42693
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:07:00.683712 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.667322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:36877 (local address 127.21.140.124:34455)
0928 21:07:00.667655 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.667667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.668648 (+ 981us) server_negotiation.cc:407] Connection header received
0928 21:07:00.669762 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.669766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.669822 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.669901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.671240 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.672294 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.673721 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.674129 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.674212 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.678842 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.678846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.678849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.678879 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.680833 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681636 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.681640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.681642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.681749 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.682222 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.682226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.682229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.682490 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.682561 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.683411 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.683540 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
I20260928 21:07:00.683709 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.669068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:38077 (local address 127.21.140.126:34327)
0928 21:07:00.669220 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.669224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.669472 (+ 248us) server_negotiation.cc:407] Connection header received
0928 21:07:00.669724 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.669727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.669780 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.669858 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.671221 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.671762 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.673751 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.674215 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.674311 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.678842 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.678846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.678849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.678879 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.681498 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.682033 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.682037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.682038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.682126 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.682430 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.682434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.682435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.682656 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.682731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.683410 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.683539 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:07:00.683722 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.667402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:52533 (local address 127.21.140.125:45677)
0928 21:07:00.667542 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.667546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.668648 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 21:07:00.669763 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.669766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.669822 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.669882 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.671240 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.672048 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.673721 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.674090 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.674188 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.679075 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.679082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.679088 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.679124 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.681004 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681657 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.681664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.681669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.681766 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.682212 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.682219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.682223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.682486 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.682606 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.683323 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.683468 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:07:00.685542 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.667841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.21.140.65:36877)
0928 21:07:00.668475 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.668537 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.669524 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.670068 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.670076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.670559 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.671079 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.671088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.672419 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.672424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.673575 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.673587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.674422 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.674425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.674687 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.675689 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.675704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.678615 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.681020 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.681049 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.681517 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.681859 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.681866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.682105 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.682646 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.682652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.683026 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.684230 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.684374 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":314,"spinlock_wait_cycles":29696,"thread_start_us":150,"threads_started":1}
I20260928 21:07:00.685556 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.667841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.21.140.65:52533)
0928 21:07:00.668475 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.668537 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.669530 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.670067 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.670076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.670559 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.671079 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.671088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.672192 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.672194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.673594 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.673600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.674421 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.674425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.674731 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.675592 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.675609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.678898 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.681140 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.681151 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.681527 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.681890 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.681901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.682099 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.682682 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.682685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.683028 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.683881 (+ 853us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.684310 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":300,"spinlock_wait_cycles":133376,"thread_start_us":132,"threads_started":1}
I20260928 21:07:00.686050 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.668961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:38077)
0928 21:07:00.669372 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.669383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.669521 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.670067 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.670073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.670487 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.671069 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.671084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.671919 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.671922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.673592 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.673606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.674513 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.674520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.674715 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.675552 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.675588 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.678615 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.681636 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.681639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.681641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.681895 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.682238 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.682241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.682243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.682315 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.682783 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.682787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.683026 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.684157 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.684438 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":299,"spinlock_wait_cycles":19328,"thread_start_us":170,"threads_started":1}
I20260928 21:07:00.686964 29157 heartbeater.cc:347] Connected to a master server at 127.21.140.125:45677
I20260928 21:07:00.687019 29156 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34327
I20260928 21:07:00.687130 29155 heartbeater.cc:347] Connected to a master server at 127.21.140.124:34455
I20260928 21:07:00.687227 29156 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:00.687227 29157 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:00.687255 29155 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:00.687897 29156 heartbeater.cc:511] Master 127.21.140.126:34327 requested a full tablet report, sending...
I20260928 21:07:00.687899 29157 heartbeater.cc:511] Master 127.21.140.125:45677 requested a full tablet report, sending...
I20260928 21:07:00.687896 29155 heartbeater.cc:511] Master 127.21.140.124:34455 requested a full tablet report, sending...
I20260928 21:07:00.689364 28182 ts_manager.cc:194] Re-registered known tserver with Master: 14a3755777754de48df817a03bf8a6d9 (127.21.140.65:40035)
I20260928 21:07:00.689527 28969 ts_manager.cc:194] Registered new tserver with Master: 14a3755777754de48df817a03bf8a6d9 (127.21.140.65:40035)
I20260928 21:07:00.689695 28782 ts_manager.cc:194] Registered new tserver with Master: 14a3755777754de48df817a03bf8a6d9 (127.21.140.65:40035)
I20260928 21:07:00.690191 28182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:38077
I20260928 21:07:00.795806 28646 heartbeater.cc:503] Master 127.21.140.126:34327 was elected leader, sending a full tablet report...
W20260928 21:07:00.800081 29163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:00.800421 29163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:00.800503 29163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:00.804695 29163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:00.804782 29163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:00.804877 29163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:00.810216 29163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:32849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=42693
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:00.811399 29163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:00.812492 29163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:00.822340 29169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.822331 29168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:00.822333 29171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:00.822482 29163 server_base.cc:1034] running on GCE node
I20260928 21:07:00.823177 29163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:00.823829 29163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:00.825009 29163 hybrid_clock.cc:648] HybridClock initialized: now 1790629620824991 us; error 43 us; skew 500 ppm
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:00.828073 29163 init.cc:377] Logged in from keytab as oryx/127.21.140.66@KRBTEST.COM (short username oryx)
I20260928 21:07:00.828984 29163 webserver.cc:460] Webserver started at http://127.21.140.66:42693/ using document root <none> and password file <none>
I20260928 21:07:00.829576 29163 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:00.829675 29163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:00.833732 29163 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:07:00.836026 29178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.837477 29163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:00.837654 29163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "aafc3b4ed0bb4412ac5d83d5724f6229"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "7dda77ea9683f3657b2544007161fbd6"
server_key_iv: "9dcf90e8d0dac6234df84e4e1f049187"
server_key_version: "encryptionkey@0"
I20260928 21:07:00.838413 29163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:00.855598 29163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:00.859222 29163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:00.859395 29163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:00.859928 29163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:00.860817 29163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:00.860873 29163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.860932 29163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:00.860950 29163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:00.869630 29163 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:32849
I20260928 21:07:00.869704 29291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:32849 every 8 connection(s)
I20260928 21:07:00.870719 29163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:00.880978 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 29163
I20260928 21:07:00.881165 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28516
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.124@KRBTEST.COM
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
Sep 28 21:07:00 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:00.887723 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.874299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:51519 (local address 127.21.140.126:34327)
0928 21:07:00.874431 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.874437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.874546 (+ 109us) server_negotiation.cc:407] Connection header received
0928 21:07:00.875138 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.875141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.875190 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.875270 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.877239 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.878065 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879411 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.879785 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879916 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.883546 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.883550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.883553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.883588 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.885180 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.885799 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.885805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.885808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.885927 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.886407 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.886414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.886419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.886685 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.886759 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.887368 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.887526 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:07:00.887748 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.872548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:42133 (local address 127.21.140.125:45677)
0928 21:07:00.872711 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.872717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.873656 (+ 939us) server_negotiation.cc:407] Connection header received
0928 21:07:00.874839 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.874842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.874918 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.875012 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.876546 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.877070 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.878677 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.878916 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879047 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.883522 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.883529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.883535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.883572 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.885503 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.886002 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.886007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.886012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.886100 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.886419 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.886425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.886430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.886686 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.886772 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.887353 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.887494 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 21:07:00.889487 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.873035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.21.140.66:42133)
0928 21:07:00.873500 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.873552 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.874634 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.875180 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.875188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.875699 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.876353 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.876368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.877215 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.877219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.878431 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.878445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.879124 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.879396 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.880152 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.880174 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.883355 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.885624 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.885628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.885630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.885867 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.886210 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.886216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.886221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.886318 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.886836 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.886838 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.887099 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.888256 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.888431 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":270,"spinlock_wait_cycles":50688,"thread_start_us":131,"threads_started":1}
I20260928 21:07:00.889497 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.874154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:51519)
0928 21:07:00.874417 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.874434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.874636 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.875324 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.875329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.875710 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.876365 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.876375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.878190 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.878197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.879294 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.879301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.879929 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.880082 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.880563 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.880580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.883355 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.885308 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.885314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.885333 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.885665 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.886089 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.886095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.886100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.886295 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.886826 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.886832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.887090 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.887907 (+ 817us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.888300 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":133,"spinlock_wait_cycles":9216,"thread_start_us":90,"threads_started":1}
I20260928 21:07:00.890112 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.872634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:36831 (local address 127.21.140.124:34455)
0928 21:07:00.872787 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:00.872793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:00.873679 (+ 886us) server_negotiation.cc:407] Connection header received
0928 21:07:00.874909 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:00.874912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:00.874983 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:00.875032 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:00.876528 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.877122 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.878651 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.879063 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879147 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.882545 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:00.882549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:00.882551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:00.882583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:00.888128 (+ 5545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.888806 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.888812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.888814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.888903 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:00.889203 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:00.889206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:00.889207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:00.889412 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:00.889498 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:00.889811 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:07:00.889944 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:07:00.890285 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:35435
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:34327
--webserver_port=43897
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:07:00.890465 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:00.873036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.21.140.66:36831)
0928 21:07:00.873501 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:00.873553 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:00.874636 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:00.875180 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:00.875188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:00.875710 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:00.876369 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.876375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.877244 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.877246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.878432 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:00.878441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:00.879306 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:00.879309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:00.879416 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:00.880219 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:00.880238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:00.882301 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:00.888447 (+ 6146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.888450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.888453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.888682 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.889022 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:00.889025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:00.889026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:00.889105 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:00.889557 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:00.889560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:00.889650 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 21:07:00.889744 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:07:00.889880 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":253,"spinlock_wait_cycles":51456,"thread_start_us":143,"threads_started":1}
I20260928 21:07:00.891433 29293 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34327
I20260928 21:07:00.891520 29294 heartbeater.cc:347] Connected to a master server at 127.21.140.125:45677
I20260928 21:07:00.891731 29294 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:00.891731 29293 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:00.892336 29294 heartbeater.cc:511] Master 127.21.140.125:45677 requested a full tablet report, sending...
I20260928 21:07:00.892340 29293 heartbeater.cc:511] Master 127.21.140.126:34327 requested a full tablet report, sending...
I20260928 21:07:00.893263 29292 heartbeater.cc:347] Connected to a master server at 127.21.140.124:34455
I20260928 21:07:00.893347 29292 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:00.893492 28182 ts_manager.cc:194] Re-registered known tserver with Master: aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849)
I20260928 21:07:00.893508 28782 ts_manager.cc:194] Registered new tserver with Master: aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849)
I20260928 21:07:00.893525 29292 heartbeater.cc:511] Master 127.21.140.124:34455 requested a full tablet report, sending...
I20260928 21:07:00.894281 28182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:51519
I20260928 21:07:00.894496 28969 ts_manager.cc:194] Registered new tserver with Master: aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849)
W20260928 21:07:01.008158 29299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:01.008481 29299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:01.008565 29299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:01.013123 29299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:01.013224 29299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:01.013324 29299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:01.019068 29299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:35435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=43897
--tserver_master_addrs=127.21.140.126:34327,127.21.140.125:45677,127.21.140.124:34455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:01.020200 29299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:01.021281 29299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:01.031553 29304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:01.031615 29299 server_base.cc:1034] running on GCE node
W20260928 21:07:01.031625 29305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:01.031528 29307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:01.032245 29299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:01.032914 29299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:01.034067 29299 hybrid_clock.cc:648] HybridClock initialized: now 1790629621034043 us; error 29 us; skew 500 ppm
Sep 28 21:07:01 dist-test-slave-d5wc krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629621, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:01.036952 29299 init.cc:377] Logged in from keytab as oryx/127.21.140.67@KRBTEST.COM (short username oryx)
I20260928 21:07:01.037912 29299 webserver.cc:460] Webserver started at http://127.21.140.67:43897/ using document root <none> and password file <none>
I20260928 21:07:01.038654 29299 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:01.038735 29299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:01.042994 29299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:07:01.045365 29314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:01.046795 29299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:01.046931 29299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "35838c8b76f145188b251f569c192617"
format_stamp: "Formatted at 2026-09-28 21:06:59 on dist-test-slave-d5wc"
server_key: "deb6be6c1d556417676a126a45b991d2"
server_key_iv: "7e3cd21c5e38e944226b67e51d4209fd"
server_key_version: "encryptionkey@0"
I20260928 21:07:01.047655 29299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:01.051049 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.046847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.21.140.124:34455)
0928 21:07:01.046991 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.046995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.047144 (+ 149us) server_negotiation.cc:407] Connection header received
0928 21:07:01.047299 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.047302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.047355 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.047427 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:01.047806 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.048399 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.049924 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.050424 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.050524 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.050645 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.050839 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.050910 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:07:01.051054 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.046715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49240)
0928 21:07:01.047035 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.047052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.047180 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.047496 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.047500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.047522 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:01.047637 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.047644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.048532 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.048537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.049787 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.049799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.050559 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.050563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.050669 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.050697 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.050784 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.050868 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
I20260928 21:07:01.051833 28977 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 21:07:01.052234 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c5bdfe48304bb3b161e60612c4a59c. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:01.052402 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: This master's current role is: LEARNER
I20260928 21:07:01.063951 29299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:01.067070 29299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:01.067286 29299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:01.067965 29299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:01.068847 29299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:01.068903 29299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:01.068977 29299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:01.069008 29299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:01.078424 29299 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35435
I20260928 21:07:01.078562 29429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35435 every 8 connection(s)
I20260928 21:07:01.079489 29299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:07:01.089097 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 29299
Sep 28 21:07:01 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629621, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.125@KRBTEST.COM
Sep 28 21:07:01 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629621, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
Sep 28 21:07:01 dist-test-slave-d5wc krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629621, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.124@KRBTEST.COM
I20260928 21:07:01.098805 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.081405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:53507 (local address 127.21.140.126:34327)
0928 21:07:01.081728 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.081733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.082793 (+ 1060us) server_negotiation.cc:407] Connection header received
0928 21:07:01.083912 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.083916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.084021 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.084072 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.085442 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.086302 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.087642 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.087942 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.088061 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.091167 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.091174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.091181 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.091216 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.096449 (+ 5233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.096993 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.096998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.097000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.097096 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.097401 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.097405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.097406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.097634 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.097739 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.098492 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.098631 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260928 21:07:01.099135 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.083279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:59971 (local address 127.21.140.125:45677)
0928 21:07:01.083396 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.083400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.083474 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:07:01.083912 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.083916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.083972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.084030 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.085451 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.086290 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.087338 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.087597 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.087704 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.091452 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.091456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.091459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.091486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.093569 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.096582 (+ 3013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.096586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.096588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.096689 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.097410 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.097414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.097416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.097889 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.098009 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.098812 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.098964 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:07:01.100060 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.082075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:53507)
0928 21:07:01.082557 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.082611 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.083731 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.084247 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.084254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.084703 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:01.085289 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.085299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.086412 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.086415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.087513 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.087523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.088063 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.088066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.088170 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.088748 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:01.088768 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:01.090916 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:01.096567 (+ 5651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:01.096570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:01.096573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:01.096868 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:01.097176 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:01.097180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:01.097183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:01.097277 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:01.097762 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:01.097773 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:01.098169 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.098429 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.098823 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":150,"spinlock_wait_cycles":20480,"thread_start_us":123,"threads_started":1}
I20260928 21:07:01.101441 29431 heartbeater.cc:347] Connected to a master server at 127.21.140.126:34327
I20260928 21:07:01.101599 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.083135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.21.140.67:59971)
0928 21:07:01.083383 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.083397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.083727 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.084247 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.084254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.084704 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:01.085289 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.085299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.086437 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.086440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.087206 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.087215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.087744 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.087748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.087939 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.088575 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:01.088590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:01.091285 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:01.094191 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:01.094198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:01.094216 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:01.094566 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:01.096813 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:01.096817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:01.096819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:01.097033 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:01.098057 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:01.098060 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:01.098171 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.098639 (+ 468us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.100871 (+ 2232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":2216,"spinlock_wait_cycles":25856,"thread_start_us":87,"threads_started":1}
I20260928 21:07:01.101682 29431 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:01.101998 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.081579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:56961 (local address 127.21.140.124:34455)
0928 21:07:01.081730 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.081734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.082791 (+ 1057us) server_negotiation.cc:407] Connection header received
0928 21:07:01.083884 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.083895 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.083957 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.084065 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.085448 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.086015 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.087652 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.087966 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.088043 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.091452 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.091456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.091459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.091486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.095639 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.099841 (+ 4202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.099845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.099850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.099953 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.100290 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.100294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.100296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.100561 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.100652 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.101714 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.101836 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:07:01.102295 29431 heartbeater.cc:511] Master 127.21.140.126:34327 requested a full tablet report, sending...
I20260928 21:07:01.102524 29435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.082076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.21.140.67:56961)
0928 21:07:01.082556 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.082612 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.083678 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.084245 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.084253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.084686 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:01.085279 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.085292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.086163 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.086166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.087514 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.087523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.088215 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.088220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.088314 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.088918 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:01.088934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:01.091287 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:01.099369 (+ 8082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:01.099374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:01.099381 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:01.099716 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:01.100066 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:01.100075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:01.100078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:01.100174 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:01.101189 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:01.101192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:01.101283 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.101375 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.101534 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":194,"spinlock_wait_cycles":23040,"thread_start_us":152,"threads_started":1}
I20260928 21:07:01.103217 29430 heartbeater.cc:347] Connected to a master server at 127.21.140.124:34455
I20260928 21:07:01.103298 29430 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:01.103480 29430 heartbeater.cc:511] Master 127.21.140.124:34455 requested a full tablet report, sending...
I20260928 21:07:01.103611 29432 heartbeater.cc:347] Connected to a master server at 127.21.140.125:45677
I20260928 21:07:01.103670 29432 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:01.103807 28182 ts_manager.cc:194] Re-registered known tserver with Master: 35838c8b76f145188b251f569c192617 (127.21.140.67:35435)
I20260928 21:07:01.103817 29432 heartbeater.cc:511] Master 127.21.140.125:45677 requested a full tablet report, sending...
I20260928 21:07:01.104148 28969 ts_manager.cc:194] Registered new tserver with Master: 35838c8b76f145188b251f569c192617 (127.21.140.67:35435)
I20260928 21:07:01.104456 28782 ts_manager.cc:194] Registered new tserver with Master: 35838c8b76f145188b251f569c192617 (127.21.140.67:35435)
I20260928 21:07:01.104657 28182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:53507
I20260928 21:07:01.105818 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.091812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.21.140.125:45677)
0928 21:07:01.093152 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.093156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.093169 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:07:01.093285 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.093288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.093335 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.093425 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.094478 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.095332 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.097362 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.097727 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.097800 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.100017 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.100021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.100023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.100048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.102308 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.102862 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.102865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.102867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.102961 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.103263 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.103267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.103269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.105093 (+ 1824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.105253 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.105540 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.105665 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":69,"threads_started":1}
I20260928 21:07:01.106930 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.092525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.21.140.124:34455)
0928 21:07:01.092785 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.092796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.093428 (+ 632us) server_negotiation.cc:407] Connection header received
0928 21:07:01.093611 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.093616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.093670 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.093737 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.094648 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.095426 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.096686 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.097059 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.097145 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.103142 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.103146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.103149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.103172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.104871 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.105645 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.105652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.105654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.105745 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.106115 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.106118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.106120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.106299 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.106453 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.106686 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.106814 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
I20260928 21:07:01.108963 29443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.091844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.21.140.126:34327)
0928 21:07:01.092094 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.092099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.092431 (+ 332us) server_negotiation.cc:407] Connection header received
0928 21:07:01.092627 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.092633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.092689 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.092759 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.093871 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.094774 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.096111 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.096500 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.096572 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.098990 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.098995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.098997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.099021 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.105380 (+ 6359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.107684 (+ 2304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.107692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.107693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.107774 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.108091 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.108094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.108096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.108285 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.108440 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.108659 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.108793 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260928 21:07:01.111683 28182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:07:01.114831 28182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:01.117620 29448 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: NON_VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: true } } }
I20260928 21:07:01.117759 29448 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:01.120814 29448 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: attempting to promote NON_VOTER 20b04c6aeb3c4c2183e04ae82b8ef6ef to VOTER
I20260928 21:07:01.121156 29448 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:01.122293 28977 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 LEARNER]: Refusing update from remote peer e5c5bdfe48304bb3b161e60612c4a59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 21:07:01.122400 28793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Refusing update from remote peer e5c5bdfe48304bb3b161e60612c4a59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 21:07:01.122673 29448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 21:07:01.122934 29315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 21:07:01.124374 29449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.124780 28977 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.125916 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.125892 29315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.126020 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:01.126035 29315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:01.126463 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.126585 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:01.127918 29315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.128032 29315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:01.128301 28793 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.130367 29457 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:01.130519 29457 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:01.137864 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.132531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35435 (local address 127.0.0.1:55154)
0928 21:07:01.132651 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.132666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.132741 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.133552 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.133555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.133572 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:01.133683 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.133691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.134948 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.134954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.135823 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.135831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.137418 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.137422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.137537 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.137561 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.137643 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.137728 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:07:01.137907 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.132692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.21.140.67:35435)
0928 21:07:01.133056 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.133069 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.133092 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:07:01.133170 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.133175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.133359 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.133584 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:01.133777 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.134831 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.136736 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.137282 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.137413 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.137570 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.137672 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.137768 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
I20260928 21:07:01.139693 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.133104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:32849 (local address 127.0.0.1:34656)
0928 21:07:01.133346 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.133361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.133535 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.134406 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.134412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.134432 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:01.134569 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.134576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.136566 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.136571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.137890 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.137896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.139314 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.139317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.139387 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.139418 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.139488 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.139538 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":21,"thread_start_us":119,"threads_started":1}
I20260928 21:07:01.139909 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.133562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.21.140.66:32849)
0928 21:07:01.133882 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.133895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.133917 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:07:01.134022 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.134029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.134245 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.134728 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:01.134866 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.136432 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.138110 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.139001 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.139105 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.139263 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.139550 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.139647 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":134,"threads_started":1}
I20260928 21:07:01.140136 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.134772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.21.140.65:40035)
0928 21:07:01.135147 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.135164 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.135190 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:07:01.135266 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.135273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.135464 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.135629 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:01.135886 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.136613 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.138218 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.138998 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.139105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.139662 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.139716 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.139919 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":90,"threads_started":1}
I20260928 21:07:01.140741 29461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.134149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:40035 (local address 127.0.0.1:59786)
0928 21:07:01.134386 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.134408 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.134495 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.135624 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.135627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.135645 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:01.135743 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.135749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.136715 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.136718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.137879 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.137890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.139255 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.139258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.139352 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.139374 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.139439 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.139532 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":28,"thread_start_us":97,"threads_started":1}
I20260928 21:07:01.141212 29364 tablet_service.cc:1467] Processing CreateTablet for tablet c8cd50326b62486089967e31e92e8c41 (DEFAULT_TABLE table=test-table [id=7ad521224fc94ae58a0612cd8ce229fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:01.142334 29364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cd50326b62486089967e31e92e8c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:01.142918 29226 tablet_service.cc:1467] Processing CreateTablet for tablet c8cd50326b62486089967e31e92e8c41 (DEFAULT_TABLE table=test-table [id=7ad521224fc94ae58a0612cd8ce229fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:01.143846 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cd50326b62486089967e31e92e8c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:01.143661 29089 tablet_service.cc:1467] Processing CreateTablet for tablet c8cd50326b62486089967e31e92e8c41 (DEFAULT_TABLE table=test-table [id=7ad521224fc94ae58a0612cd8ce229fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:01.144623 29089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cd50326b62486089967e31e92e8c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:01.150789 29464 tablet_bootstrap.cc:492] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Bootstrap starting.
I20260928 21:07:01.151074 29463 tablet_bootstrap.cc:492] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Bootstrap starting.
I20260928 21:07:01.152320 29465 tablet_bootstrap.cc:492] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Bootstrap starting.
I20260928 21:07:01.153339 29463 tablet_bootstrap.cc:654] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:01.153599 29464 tablet_bootstrap.cc:654] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:01.154317 29463 log.cc:824] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:01.154342 29464 log.cc:824] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:01.154762 29465 tablet_bootstrap.cc:654] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:01.155581 29465 log.cc:824] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:01.156188 29464 tablet_bootstrap.cc:492] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: No bootstrap required, opened a new log
I20260928 21:07:01.156407 29464 ts_tablet_manager.cc:1397] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:07:01.157630 29465 tablet_bootstrap.cc:492] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: No bootstrap required, opened a new log
I20260928 21:07:01.157877 29465 ts_tablet_manager.cc:1397] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:07:01.160247 29464 raft_consensus.cc:348] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.160548 29464 raft_consensus.cc:374] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:01.160607 29464 raft_consensus.cc:729] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35838c8b76f145188b251f569c192617, State: Initialized, Role: FOLLOWER
I20260928 21:07:01.161091 29464 consensus_queue.cc:260] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.161767 29465 raft_consensus.cc:348] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.162004 29465 raft_consensus.cc:374] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:01.162107 29465 raft_consensus.cc:729] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a3755777754de48df817a03bf8a6d9, State: Initialized, Role: FOLLOWER
I20260928 21:07:01.162130 29464 ts_tablet_manager.cc:1428] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:07:01.162358 29431 heartbeater.cc:503] Master 127.21.140.126:34327 was elected leader, sending a full tablet report...
I20260928 21:07:01.162598 29465 consensus_queue.cc:260] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.163514 29465 ts_tablet_manager.cc:1428] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260928 21:07:01.163621 29463 tablet_bootstrap.cc:492] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: No bootstrap required, opened a new log
I20260928 21:07:01.163806 29463 ts_tablet_manager.cc:1397] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260928 21:07:01.165729 29156 heartbeater.cc:503] Master 127.21.140.126:34327 was elected leader, sending a full tablet report...
W20260928 21:07:01.167363 29158 tablet.cc:2369] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:07:01.168956 29463 raft_consensus.cc:348] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.169279 29463 raft_consensus.cc:374] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:01.169348 29463 raft_consensus.cc:729] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aafc3b4ed0bb4412ac5d83d5724f6229, State: Initialized, Role: FOLLOWER
I20260928 21:07:01.169760 29463 consensus_queue.cc:260] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.170468 29293 heartbeater.cc:503] Master 127.21.140.126:34327 was elected leader, sending a full tablet report...
I20260928 21:07:01.170775 29463 ts_tablet_manager.cc:1428] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260928 21:07:01.302683 29470 raft_consensus.cc:482] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:01.302943 29470 raft_consensus.cc:504] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.304138 29470 leader_election.cc:288] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35838c8b76f145188b251f569c192617 (127.21.140.67:35435), aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849)
I20260928 21:07:01.308295 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.304557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:57949 (local address 127.21.140.66:32849)
0928 21:07:01.304693 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.304700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.304870 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:07:01.305094 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.305097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.305146 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.305237 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:01.305502 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.306058 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.307268 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.307664 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.307802 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.307858 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.308089 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.308160 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:07:01.308300 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.304568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:45241 (local address 127.21.140.67:35435)
0928 21:07:01.304703 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.304709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.305040 (+ 331us) server_negotiation.cc:407] Connection header received
0928 21:07:01.305285 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.305288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.305344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.305432 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:01.305683 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.306289 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.307291 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.307677 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.307812 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.307867 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.308090 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.308165 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:07:01.308307 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.304442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:32849 (local address 127.21.140.65:57949)
0928 21:07:01.304773 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.304786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.304970 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.305250 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.305253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.305270 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:01.305384 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.305390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.306176 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.306181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.307145 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.307154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.307802 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.307806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.307970 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.307987 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.308073 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.308160 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":53,"thread_start_us":92,"threads_started":1}
I20260928 21:07:01.308317 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.304442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35435 (local address 127.21.140.65:45241)
0928 21:07:01.304954 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:01.304969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:01.305075 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:01.305445 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:01.305448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:01.305465 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:01.305563 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.305574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.306402 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.306407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.307165 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:01.307173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.307804 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.307808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:01.307958 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.307982 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:07:01.308073 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:07:01.308165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":316,"thread_start_us":103,"threads_started":1}
I20260928 21:07:01.309007 29384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cd50326b62486089967e31e92e8c41" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35838c8b76f145188b251f569c192617" is_pre_election: true
I20260928 21:07:01.309033 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cd50326b62486089967e31e92e8c41" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" is_pre_election: true
I20260928 21:07:01.309322 29246 raft_consensus.cc:2454] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 0.
I20260928 21:07:01.309321 29384 raft_consensus.cc:2454] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 0.
I20260928 21:07:01.309779 29042 leader_election.cc:302] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a3755777754de48df817a03bf8a6d9, aafc3b4ed0bb4412ac5d83d5724f6229; no voters:
I20260928 21:07:01.310133 29470 raft_consensus.cc:2781] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:01.310201 29470 raft_consensus.cc:482] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:01.310253 29470 raft_consensus.cc:3037] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:01.311323 29470 raft_consensus.cc:504] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.311695 29470 leader_election.cc:288] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 election: Requested vote from peers 35838c8b76f145188b251f569c192617 (127.21.140.67:35435), aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849)
I20260928 21:07:01.312064 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cd50326b62486089967e31e92e8c41" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229"
I20260928 21:07:01.312127 29384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cd50326b62486089967e31e92e8c41" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35838c8b76f145188b251f569c192617"
I20260928 21:07:01.312209 29246 raft_consensus.cc:3037] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:01.312235 29384 raft_consensus.cc:3037] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:01.313529 29384 raft_consensus.cc:2454] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 1.
I20260928 21:07:01.313529 29246 raft_consensus.cc:2454] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 1.
I20260928 21:07:01.313858 29045 leader_election.cc:302] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a3755777754de48df817a03bf8a6d9, 35838c8b76f145188b251f569c192617; no voters:
I20260928 21:07:01.314113 29470 raft_consensus.cc:2781] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:01.314309 29470 raft_consensus.cc:686] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 1 LEADER]: Becoming Leader. State: Replica: 14a3755777754de48df817a03bf8a6d9, State: Running, Role: LEADER
I20260928 21:07:01.314675 29470 consensus_queue.cc:237] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } }
I20260928 21:07:01.318250 28182 catalog_manager.cc:5495] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14a3755777754de48df817a03bf8a6d9 (127.21.140.65). New cstate: current_term: 1 leader_uuid: "14a3755777754de48df817a03bf8a6d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } health_report { overall_health: HEALTHY } } }
W20260928 21:07:01.331324 29433 tablet.cc:2369] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:07:01.334066 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.330360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.21.140.65:40035)
0928 21:07:01.330500 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.330505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.330520 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:07:01.330648 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.330650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.330704 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.330811 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:01.331054 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.331775 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.332624 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.332829 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.332909 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.333214 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:01.333619 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:01.333708 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.333813 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.333865 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:07:01.339882 29246 raft_consensus.cc:1261] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Refusing update from remote peer 14a3755777754de48df817a03bf8a6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:01.339877 29384 raft_consensus.cc:1261] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Refusing update from remote peer 14a3755777754de48df817a03bf8a6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:01.340576 29470 consensus_queue.cc:1035] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:01.340749 29474 consensus_queue.cc:1035] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:01.351189 29479 mvcc.cc:205] Tried to move back new op lower bound from 7334418929001033728 to 7334418928907673600. Current Snapshot: MvccSnapshot[applied={T|T < 7334418929001033728}]
I20260928 21:07:01.361850 28181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:01.362076 28181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:01.366715 28181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5c5bdfe48304bb3b161e60612c4a59c: Sending DeleteTablet for 3 replicas of tablet c8cd50326b62486089967e31e92e8c41
I20260928 21:07:01.367538 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet c8cd50326b62486089967e31e92e8c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:01 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:34656
I20260928 21:07:01.367686 29364 tablet_service.cc:1514] Processing DeleteTablet for tablet c8cd50326b62486089967e31e92e8c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:01 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:55154
I20260928 21:07:01.367681 29089 tablet_service.cc:1514] Processing DeleteTablet for tablet c8cd50326b62486089967e31e92e8c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:01 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:59786
I20260928 21:07:01.368110 29486 tablet_replica.cc:330] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: stopping tablet replica
I20260928 21:07:01.368384 29488 tablet_replica.cc:330] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: stopping tablet replica
I20260928 21:07:01.368458 29486 raft_consensus.cc:2229] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:01.368649 29488 raft_consensus.cc:2229] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:01.368767 29486 raft_consensus.cc:2258] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:01.368875 29488 raft_consensus.cc:2258] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:01.370237 29487 tablet_replica.cc:330] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: stopping tablet replica
I20260928 21:07:01.370532 29487 raft_consensus.cc:2229] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:07:01.370692 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28150
I20260928 21:07:01.370934 29487 raft_consensus.cc:2258] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:01.371592 29488 ts_tablet_manager.cc:1905] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:01.372836 29487 ts_tablet_manager.cc:1905] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:01.375241 29488 ts_tablet_manager.cc:1918] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:01.375352 29488 log.cc:1196] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/c8cd50326b62486089967e31e92e8c41
I20260928 21:07:01.375711 29488 ts_tablet_manager.cc:1939] T c8cd50326b62486089967e31e92e8c41 P 35838c8b76f145188b251f569c192617: Deleting consensus metadata
I20260928 21:07:01.375852 29486 ts_tablet_manager.cc:1905] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:01.379040 29486 ts_tablet_manager.cc:1918] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:01.379145 29486 log.cc:1196] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/c8cd50326b62486089967e31e92e8c41
W20260928 21:07:01.379423 29319 connection.cc:535] server connection from 127.0.0.1:55154 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:07:01.379509 29486 ts_tablet_manager.cc:1939] T c8cd50326b62486089967e31e92e8c41 P aafc3b4ed0bb4412ac5d83d5724f6229: Deleting consensus metadata
I20260928 21:07:01.380338 29487 ts_tablet_manager.cc:1918] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:01.380514 29487 log.cc:1196] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/c8cd50326b62486089967e31e92e8c41
I20260928 21:07:01.380875 29487 ts_tablet_manager.cc:1939] T c8cd50326b62486089967e31e92e8c41 P 14a3755777754de48df817a03bf8a6d9: Deleting consensus metadata
W20260928 21:07:01.381910 29043 connection.cc:441] server connection from 127.0.0.1:59786 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59786 (request call id 1) could send its response
W20260928 21:07:01.383245 29181 connection.cc:441] server connection from 127.0.0.1:34656 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34656 (request call id 1) could send its response
I20260928 21:07:01.394141 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.369234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.21.140.125:45677)
0928 21:07:01.369726 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.369731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.369745 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:01.369781 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.369783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.369828 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.369902 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.384513 (+ 14611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.385378 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.386802 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.387190 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.387280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.389554 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.389560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.389562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.389589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.391972 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.392559 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.392563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.392565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.392677 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.393056 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.393062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.393064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.393277 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.393383 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.393761 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.393889 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260928 21:07:01.394562 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.384061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.21.140.125:45677)
0928 21:07:01.384174 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.384178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.384472 (+ 294us) server_negotiation.cc:407] Connection header received
0928 21:07:01.384639 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.384641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.384689 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.384739 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.385659 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.386456 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.387642 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.387977 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.388075 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.389683 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.389687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.389690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.389716 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.392038 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.392580 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.392584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.392586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.392694 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.392976 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.392981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.392986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.393211 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.393337 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.393867 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.394065 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:07:01.395560 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.384654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.21.140.124:34455)
0928 21:07:01.385560 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.385567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.385831 (+ 264us) server_negotiation.cc:407] Connection header received
0928 21:07:01.386054 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.386058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.386111 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.386165 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.387540 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.388323 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.389575 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.389995 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.390102 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.391831 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.391838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.391843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.391876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.393625 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.394112 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.394115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.394117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.394200 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.394593 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.394598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.394600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.394807 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.394906 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.395263 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.395379 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814}
I20260928 21:07:01.403731 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:01.369901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.21.140.124:34455)
0928 21:07:01.370029 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:01.370034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:01.395472 (+ 25438us) server_negotiation.cc:407] Connection header received
0928 21:07:01.395646 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:01.395649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:01.395700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:01.395784 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:01.396571 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.397361 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.398267 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:01.398553 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:01.398655 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:01.400466 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:01.400470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:01.400472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:01.400496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:01.402097 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.402536 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.402539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.402541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.402623 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:01.402891 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:01.402896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:01.402898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:01.403081 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:01.403215 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:01.403448 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 21:07:01.403566 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:07:02.355964 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.355348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:45585)
0928 21:07:02.355686 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.355816 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
I20260928 21:07:02.356694 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.356215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:38889)
0928 21:07:02.356480 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.356582 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260928 21:07:02.356681 29431 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:34327 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.356916 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.355730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:53583)
0928 21:07:02.356579 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.356670 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":97,"threads_started":1}
W20260928 21:07:02.356976 29293 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:34327 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
W20260928 21:07:02.357174 29156 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:34327 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.867027 29500 raft_consensus.cc:482] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e5c5bdfe48304bb3b161e60612c4a59c)
I20260928 21:07:02.867031 29499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e5c5bdfe48304bb3b161e60612c4a59c)
I20260928 21:07:02.867314 29500 raft_consensus.cc:504] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:02.867316 29499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:02.868568 29499 leader_election.cc:288] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327), da816907c57947f0aab768c023fbe692 (127.21.140.125:45677)
I20260928 21:07:02.869002 29500 leader_election.cc:288] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327), 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124:34455)
I20260928 21:07:02.869438 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.868743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50294)
0928 21:07:02.869179 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.869291 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":85,"threads_started":1}
W20260928 21:07:02.869670 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.870240 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.868750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50292)
0928 21:07:02.869998 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.870102 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
W20260928 21:07:02.870451 28776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.871695 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.871378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50308)
0928 21:07:02.871539 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.871599 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 21:07:02.871863 28776 leader_election.cc:334] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.873476 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.873160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50320)
0928 21:07:02.873308 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.873371 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 21:07:02.873674 28961 leader_election.cc:334] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.874770 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.869514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.21.140.124:34455)
0928 21:07:02.869931 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:02.869935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:02.869949 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:02.870026 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:02.870029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:02.870078 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:02.870167 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:02.870660 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.871553 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.873205 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.873946 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.874106 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:02.874173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:02.874542 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 21:07:02.874646 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":100,"threads_started":1}
I20260928 21:07:02.874827 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.868743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34455 (local address 127.0.0.1:49286)
0928 21:07:02.869576 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.869594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:02.869743 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:02.870350 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:02.870354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:02.870375 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:02.870502 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:02.870510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.871693 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.871698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:02.873092 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:02.873104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.874123 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.874126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:02.874343 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:02.874396 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 21:07:02.874500 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:07:02.874623 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":37,"thread_start_us":115,"threads_started":1}
I20260928 21:07:02.875087 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.869160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:45677 (local address 127.0.0.1:55158)
0928 21:07:02.869431 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.869447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:02.869575 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:02.871029 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:02.871032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:02.871057 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:02.871199 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:02.871209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.872367 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.872374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:02.873605 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:02.873620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.874590 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.874595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:02.874704 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:02.874733 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 21:07:02.874823 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:07:02.874952 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":64,"threads_started":1}
I20260928 21:07:02.875285 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.869263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.21.140.125:45677)
0928 21:07:02.870722 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:02.870731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:02.870753 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:07:02.870806 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:02.870809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:02.870863 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:02.871336 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:02.871452 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.872219 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.873748 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:02.874441 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:02.874585 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:02.874628 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:02.874854 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:07:02.874949 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":1376,"thread_start_us":90,"threads_started":1}
I20260928 21:07:02.875815 28793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da816907c57947f0aab768c023fbe692" is_pre_election: true
I20260928 21:07:02.875813 28977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da816907c57947f0aab768c023fbe692" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" is_pre_election: true
I20260928 21:07:02.876111 28793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b04c6aeb3c4c2183e04ae82b8ef6ef in term 1.
I20260928 21:07:02.876111 28977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da816907c57947f0aab768c023fbe692 in term 1.
I20260928 21:07:02.876549 28777 leader_election.cc:302] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b04c6aeb3c4c2183e04ae82b8ef6ef, da816907c57947f0aab768c023fbe692; no voters: e5c5bdfe48304bb3b161e60612c4a59c
I20260928 21:07:02.876601 28961 leader_election.cc:302] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b04c6aeb3c4c2183e04ae82b8ef6ef, da816907c57947f0aab768c023fbe692; no voters: e5c5bdfe48304bb3b161e60612c4a59c
I20260928 21:07:02.876863 29499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:07:02.876863 29500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:07:02.876929 29499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Starting leader election (detected failure of leader e5c5bdfe48304bb3b161e60612c4a59c)
I20260928 21:07:02.876929 29500 raft_consensus.cc:482] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e5c5bdfe48304bb3b161e60612c4a59c)
I20260928 21:07:02.876976 29500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:07:02.876976 29499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:07:02.878602 29499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:02.878602 29500 raft_consensus.cc:504] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:02.879104 29499 leader_election.cc:288] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [CANDIDATE]: Term 2 election: Requested vote from peers e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327), da816907c57947f0aab768c023fbe692 (127.21.140.125:45677)
I20260928 21:07:02.879118 29500 leader_election.cc:288] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 2 election: Requested vote from peers e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327), 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124:34455)
I20260928 21:07:02.879627 28977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da816907c57947f0aab768c023fbe692" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
I20260928 21:07:02.879739 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.879386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50324)
0928 21:07:02.879553 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.879628 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 21:07:02.879837 28977 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate da816907c57947f0aab768c023fbe692 in current term 2: Already voted for candidate 20b04c6aeb3c4c2183e04ae82b8ef6ef in this term.
I20260928 21:07:02.879963 28793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da816907c57947f0aab768c023fbe692"
I20260928 21:07:02.880102 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.879502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50332)
0928 21:07:02.879776 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.879839 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260928 21:07:02.880134 28793 raft_consensus.cc:2379] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 20b04c6aeb3c4c2183e04ae82b8ef6ef in current term 2: Already voted for candidate da816907c57947f0aab768c023fbe692 in this term.
I20260928 21:07:02.880792 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.880302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50342)
0928 21:07:02.880609 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.880678 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260928 21:07:02.881021 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:02.880736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:50356)
0928 21:07:02.880895 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:02.880954 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 21:07:02.880995 28776 leader_election.cc:334] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.881083 28776 leader_election.cc:302] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da816907c57947f0aab768c023fbe692; no voters: 20b04c6aeb3c4c2183e04ae82b8ef6ef, e5c5bdfe48304bb3b161e60612c4a59c
W20260928 21:07:02.881201 28961 leader_election.cc:334] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:02.881276 28961 leader_election.cc:302] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b04c6aeb3c4c2183e04ae82b8ef6ef; no voters: da816907c57947f0aab768c023fbe692, e5c5bdfe48304bb3b161e60612c4a59c
I20260928 21:07:02.881321 29500 raft_consensus.cc:2726] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:07:02.881412 29499 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:07:03.358225 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:03.357567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:55271)
0928 21:07:03.357992 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:03.358085 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260928 21:07:03.358225 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:03.357565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:33985)
0928 21:07:03.358000 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:03.358107 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":145,"threads_started":1}
I20260928 21:07:03.358827 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:03.358003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:59911)
0928 21:07:03.358629 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:03.358708 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":97,"threads_started":1}
I20260928 21:07:04.359846 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.359336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:46283)
0928 21:07:04.359628 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.359704 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":63,"threads_started":1}
I20260928 21:07:04.359912 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.359030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:49731)
0928 21:07:04.359718 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.359789 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":123,"threads_started":1}
I20260928 21:07:04.360078 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.359499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:33199)
0928 21:07:04.359889 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.359965 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":75,"threads_started":1}
I20260928 21:07:04.830144 29515 raft_consensus.cc:482] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:04.830286 29515 raft_consensus.cc:504] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:04.830704 29515 leader_election.cc:288] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327), 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124:34455)
I20260928 21:07:04.831213 28977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da816907c57947f0aab768c023fbe692" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" is_pre_election: true
I20260928 21:07:04.831380 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.830942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:60034)
0928 21:07:04.831184 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.831241 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260928 21:07:04.831386 28977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da816907c57947f0aab768c023fbe692 in term 2.
I20260928 21:07:04.831713 28777 leader_election.cc:302] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b04c6aeb3c4c2183e04ae82b8ef6ef, da816907c57947f0aab768c023fbe692; no voters:
I20260928 21:07:04.831925 29515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 21:07:04.832001 29515 raft_consensus.cc:482] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:04.832044 29515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:07:04.832264 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.832036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:60046)
0928 21:07:04.832157 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.832214 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:07:04.832365 28776 leader_election.cc:334] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:04.833273 29515 raft_consensus.cc:504] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:04.833613 29515 leader_election.cc:288] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 3 election: Requested vote from peers e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327), 20b04c6aeb3c4c2183e04ae82b8ef6ef (127.21.140.124:34455)
I20260928 21:07:04.834043 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.833765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:60060)
0928 21:07:04.833857 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.833901 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:07:04.834035 28977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da816907c57947f0aab768c023fbe692" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef"
I20260928 21:07:04.834162 28977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:07:04.835199 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.835011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:60076)
0928 21:07:04.835107 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.835152 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:07:04.835367 28776 leader_election.cc:334] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
I20260928 21:07:04.835592 28977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da816907c57947f0aab768c023fbe692 in term 3.
I20260928 21:07:04.835858 28777 leader_election.cc:302] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b04c6aeb3c4c2183e04ae82b8ef6ef, da816907c57947f0aab768c023fbe692; no voters: e5c5bdfe48304bb3b161e60612c4a59c
I20260928 21:07:04.836087 29515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:07:04.836280 29515 raft_consensus.cc:686] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [term 3 LEADER]: Becoming Leader. State: Replica: da816907c57947f0aab768c023fbe692, State: Running, Role: LEADER
I20260928 21:07:04.836602 29515 consensus_queue.cc:237] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } }
I20260928 21:07:04.837714 29518 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da816907c57947f0aab768c023fbe692. Latest consensus state: current_term: 3 leader_uuid: "da816907c57947f0aab768c023fbe692" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:04.837855 29518 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:04.838294 29520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:04.840566 29520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:04.840989 29520 catalog_manager.cc:1259] Loaded cluster ID: 0ca16b2057d44233b2d9ff3007c837fe
I20260928 21:07:04.841035 29520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:04.841625 29520 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:04.842093 29520 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Loaded TSK: 0
I20260928 21:07:04.842846 29520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:07:04.882056 28782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:07:04.885385 28782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:04.888675 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.888432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:60088)
0928 21:07:04.888553 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.888605 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:07:04.888700 28977 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [term 3 FOLLOWER]: Refusing update from remote peer da816907c57947f0aab768c023fbe692: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 21:07:04.889225 29515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 21:07:04.889679 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.889354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.0.0.1:60098)
0928 21:07:04.889522 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.889585 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 21:07:04.890002 28776 consensus_peers.cc:487] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 -> Peer e5c5bdfe48304bb3b161e60612c4a59c (127.21.140.126:34327): Couldn't send request to peer e5c5bdfe48304bb3b161e60612c4a59c. Status: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:07:04.890846 29522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader da816907c57947f0aab768c023fbe692. Latest consensus state: current_term: 3 leader_uuid: "da816907c57947f0aab768c023fbe692" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:04.890990 29522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:04.892062 29518 sys_catalog.cc:455] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "da816907c57947f0aab768c023fbe692" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:04.892190 29518 sys_catalog.cc:458] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:04.892443 29522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "da816907c57947f0aab768c023fbe692" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e5c5bdfe48304bb3b161e60612c4a59c" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 34327 } } peers { permanent_uuid: "da816907c57947f0aab768c023fbe692" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 45677 } attrs { promote: false } } peers { permanent_uuid: "20b04c6aeb3c4c2183e04ae82b8ef6ef" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34455 } attrs { promote: false } } }
I20260928 21:07:04.892570 29522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b04c6aeb3c4c2183e04ae82b8ef6ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:07:04.903635 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.898510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:40035 (local address 127.0.0.1:35892)
0928 21:07:04.898616 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.898633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:04.898736 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:04.899736 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:04.899740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:04.899759 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:04.899929 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:04.899938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.901063 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.901070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:04.902324 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:04.902334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.903194 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.903198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:04.903309 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:04.903336 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:07:04.903421 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:07:04.903494 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:07:04.903697 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.898820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.21.140.65:40035)
0928 21:07:04.899382 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:04.899387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:04.899406 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:07:04.899539 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:04.899542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:04.899596 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:04.900073 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:04.900199 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.900940 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.902443 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.903053 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.903164 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:04.903228 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:04.903463 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 21:07:04.903562 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":87,"threads_started":1}
I20260928 21:07:04.904105 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.898778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:32849 (local address 127.0.0.1:53584)
0928 21:07:04.899458 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.899475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:04.899565 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:04.900115 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:04.900121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:04.900153 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:04.900372 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:04.900381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.901514 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.901518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:04.902670 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:04.902678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.903691 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.903694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:04.903789 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:04.903816 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:07:04.903888 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:07:04.903971 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":69,"spinlock_wait_cycles":5376,"thread_start_us":91,"threads_started":1}
I20260928 21:07:04.904450 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.899075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.21.140.66:32849)
0928 21:07:04.899857 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:04.899862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:04.899875 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:07:04.899922 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:04.899926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:04.899985 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:04.900189 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:04.900549 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.901390 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.902834 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.903482 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.903580 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:04.903636 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:04.904256 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 21:07:04.904332 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":76,"threads_started":1}
I20260928 21:07:04.904446 29089 tablet_service.cc:1467] Processing CreateTablet for tablet b65132251435417ca88a258b567e0cb2 (DEFAULT_TABLE table=test-table [id=cd7e917c9cae49d68a516f80ec193fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:04.904794 29089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b65132251435417ca88a258b567e0cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:04.906626 29226 tablet_service.cc:1467] Processing CreateTablet for tablet b65132251435417ca88a258b567e0cb2 (DEFAULT_TABLE table=test-table [id=cd7e917c9cae49d68a516f80ec193fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:04.906847 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.899296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.21.140.67:35435)
0928 21:07:04.899780 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:04.899784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:04.899801 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:07:04.899855 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:04.899858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:04.899921 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:04.900018 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:04.900426 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.901372 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.903966 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.904589 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.904685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:04.904733 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:04.906603 (+ 1870us) server_negotiation.cc:299] Negotiation successful
0928 21:07:04.906710 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":99,"threads_started":1}
I20260928 21:07:04.906925 29226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b65132251435417ca88a258b567e0cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:04.907050 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.899158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35435 (local address 127.0.0.1:34550)
0928 21:07:04.899466 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.899483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:04.899577 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:04.900116 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:04.900119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:04.900151 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:04.900291 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:04.900297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.902771 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.902773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:04.903845 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:04.903860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:04.906288 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:04.906295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:04.906429 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:04.906465 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:07:04.906559 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:07:04.906647 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":10,"spinlock_wait_cycles":8448,"thread_start_us":106,"threads_started":1}
I20260928 21:07:04.907723 29534 tablet_bootstrap.cc:492] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9: Bootstrap starting.
I20260928 21:07:04.908749 29534 tablet_bootstrap.cc:654] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:04.909510 29535 tablet_bootstrap.cc:492] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Bootstrap starting.
I20260928 21:07:04.909879 29364 tablet_service.cc:1467] Processing CreateTablet for tablet b65132251435417ca88a258b567e0cb2 (DEFAULT_TABLE table=test-table [id=cd7e917c9cae49d68a516f80ec193fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:04.910202 29364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b65132251435417ca88a258b567e0cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:04.910470 29535 tablet_bootstrap.cc:654] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:04.910800 29534 tablet_bootstrap.cc:492] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9: No bootstrap required, opened a new log
I20260928 21:07:04.910890 29534 ts_tablet_manager.cc:1397] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:04.911306 29534 raft_consensus.cc:348] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:04.911439 29534 raft_consensus.cc:374] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:04.911495 29534 raft_consensus.cc:729] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a3755777754de48df817a03bf8a6d9, State: Initialized, Role: FOLLOWER
I20260928 21:07:04.911684 29534 consensus_queue.cc:260] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:04.911903 29535 tablet_bootstrap.cc:492] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: No bootstrap required, opened a new log
I20260928 21:07:04.912048 29534 ts_tablet_manager.cc:1428] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:04.913107 29539 tablet_bootstrap.cc:492] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Bootstrap starting.
I20260928 21:07:04.913909 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.913094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:34833)
0928 21:07:04.913750 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.913819 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":112,"threads_started":1}
I20260928 21:07:04.914165 29539 tablet_bootstrap.cc:654] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:04.912004 29535 ts_tablet_manager.cc:1397] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:04.915275 29535 raft_consensus.cc:348] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:04.915387 29535 raft_consensus.cc:374] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:04.915413 29535 raft_consensus.cc:729] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aafc3b4ed0bb4412ac5d83d5724f6229, State: Initialized, Role: FOLLOWER
I20260928 21:07:04.915577 29535 consensus_queue.cc:260] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:04.915954 29535 ts_tablet_manager.cc:1428] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:04.916857 29539 tablet_bootstrap.cc:492] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: No bootstrap required, opened a new log
I20260928 21:07:04.916950 29539 ts_tablet_manager.cc:1397] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:07:04.917469 29539 raft_consensus.cc:348] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:04.917613 29539 raft_consensus.cc:374] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:04.917656 29539 raft_consensus.cc:729] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35838c8b76f145188b251f569c192617, State: Initialized, Role: FOLLOWER
I20260928 21:07:04.917815 29539 consensus_queue.cc:260] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:04.917884 29294 heartbeater.cc:503] Master 127.21.140.125:45677 was elected leader, sending a full tablet report...
I20260928 21:07:04.918186 29539 ts_tablet_manager.cc:1428] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:04.918757 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.916263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:58401)
0928 21:07:04.918552 (+ 2289us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.918640 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2192,"thread_start_us":93,"threads_started":1}
I20260928 21:07:04.919551 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.919241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:38811)
0928 21:07:04.919368 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.919468 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:07:04.920039 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.918621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:60849)
0928 21:07:04.919830 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.919884 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":99,"threads_started":1}
I20260928 21:07:04.920377 29432 heartbeater.cc:503] Master 127.21.140.125:45677 was elected leader, sending a full tablet report...
I20260928 21:07:04.920717 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:04.920506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:47793)
0928 21:07:04.920625 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:04.920668 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:07:05.109239 29538 raft_consensus.cc:482] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:05.109411 29538 raft_consensus.cc:504] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:05.109863 29538 leader_election.cc:288] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849), 35838c8b76f145188b251f569c192617 (127.21.140.67:35435)
I20260928 21:07:05.110414 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b65132251435417ca88a258b567e0cb2" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" is_pre_election: true
I20260928 21:07:05.110472 29384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b65132251435417ca88a258b567e0cb2" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35838c8b76f145188b251f569c192617" is_pre_election: true
I20260928 21:07:05.110597 29246 raft_consensus.cc:2454] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 0.
I20260928 21:07:05.110656 29384 raft_consensus.cc:2454] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 0.
I20260928 21:07:05.111049 29042 leader_election.cc:302] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a3755777754de48df817a03bf8a6d9, aafc3b4ed0bb4412ac5d83d5724f6229; no voters:
I20260928 21:07:05.111238 29538 raft_consensus.cc:2781] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:05.111297 29538 raft_consensus.cc:482] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:05.111336 29538 raft_consensus.cc:3037] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:05.112454 29538 raft_consensus.cc:504] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:05.112803 29538 leader_election.cc:288] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 election: Requested vote from peers aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849), 35838c8b76f145188b251f569c192617 (127.21.140.67:35435)
I20260928 21:07:05.113157 29246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b65132251435417ca88a258b567e0cb2" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229"
I20260928 21:07:05.113183 29384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b65132251435417ca88a258b567e0cb2" candidate_uuid: "14a3755777754de48df817a03bf8a6d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35838c8b76f145188b251f569c192617"
I20260928 21:07:05.113252 29246 raft_consensus.cc:3037] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:05.113329 29384 raft_consensus.cc:3037] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:05.114289 29246 raft_consensus.cc:2454] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 1.
I20260928 21:07:05.114317 29384 raft_consensus.cc:2454] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a3755777754de48df817a03bf8a6d9 in term 1.
I20260928 21:07:05.114634 29042 leader_election.cc:302] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a3755777754de48df817a03bf8a6d9, aafc3b4ed0bb4412ac5d83d5724f6229; no voters:
I20260928 21:07:05.114831 29538 raft_consensus.cc:2781] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:05.115087 29538 raft_consensus.cc:686] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [term 1 LEADER]: Becoming Leader. State: Replica: 14a3755777754de48df817a03bf8a6d9, State: Running, Role: LEADER
I20260928 21:07:05.115268 29538 consensus_queue.cc:237] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } }
I20260928 21:07:05.115466 29157 heartbeater.cc:503] Master 127.21.140.125:45677 was elected leader, sending a full tablet report...
I20260928 21:07:05.116156 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.115850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:60291)
0928 21:07:05.116014 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.116078 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 21:07:05.117020 28782 catalog_manager.cc:5495] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14a3755777754de48df817a03bf8a6d9 (127.21.140.65). New cstate: current_term: 1 leader_uuid: "14a3755777754de48df817a03bf8a6d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a3755777754de48df817a03bf8a6d9" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 40035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 } health_report { overall_health: UNKNOWN } } }
W20260928 21:07:05.125190 29295 tablet.cc:2369] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:07:05.143875 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.140546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.21.140.65:40035)
0928 21:07:05.140669 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:05.140673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:05.140688 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:07:05.140794 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:05.140797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:05.140852 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:05.140968 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:05.141215 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:05.141743 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:05.142810 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:05.143048 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:05.143164 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:05.143417 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:05.143503 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:05.143587 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:05.143681 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:07:05.143733 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:07:05.146245 29246 raft_consensus.cc:1261] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Refusing update from remote peer 14a3755777754de48df817a03bf8a6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:05.146400 29384 raft_consensus.cc:1261] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Refusing update from remote peer 14a3755777754de48df817a03bf8a6d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:05.146839 29546 consensus_queue.cc:1035] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aafc3b4ed0bb4412ac5d83d5724f6229" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:05.147002 29538 consensus_queue.cc:1035] T b65132251435417ca88a258b567e0cb2 P 14a3755777754de48df817a03bf8a6d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35838c8b76f145188b251f569c192617" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:05.148798 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.148413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:47921)
0928 21:07:05.148632 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.148702 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 21:07:05.149108 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.148751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:60285)
0928 21:07:05.148982 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.149036 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 21:07:05.152093 29552 mvcc.cc:205] Tried to move back new op lower bound from 7334418944593903616 to 7334418944473444352. Current Snapshot: MvccSnapshot[applied={T|T < 7334418944593903616}]
I20260928 21:07:05.152695 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.152419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.65:36531)
0928 21:07:05.152505 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.152615 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:07:05.154359 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.154059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.67:34661)
0928 21:07:05.154237 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.154289 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 21:07:05.154889 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.153126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:34327 (local address 127.21.140.66:60931)
0928 21:07:05.154767 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.154820 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1587,"mutex_wait_us":1520}
I20260928 21:07:05.157656 28783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:05.157784 28783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:05.161396 28783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da816907c57947f0aab768c023fbe692: Sending DeleteTablet for 3 replicas of tablet b65132251435417ca88a258b567e0cb2
I20260928 21:07:05.162137 29089 tablet_service.cc:1514] Processing DeleteTablet for tablet b65132251435417ca88a258b567e0cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:05 UTC) from {username='oryx', principal='oryx/127.21.140.125@KRBTEST.COM'} at 127.0.0.1:35892
I20260928 21:07:05.162154 29226 tablet_service.cc:1514] Processing DeleteTablet for tablet b65132251435417ca88a258b567e0cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:05 UTC) from {username='oryx', principal='oryx/127.21.140.125@KRBTEST.COM'} at 127.0.0.1:53584
I20260928 21:07:05.162531 29558 tablet_replica.cc:330] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: stopping tablet replica
I20260928 21:07:05.162652 29558 raft_consensus.cc:2229] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:05.162809 29558 raft_consensus.cc:2258] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:05.162775 29364 tablet_service.cc:1514] Processing DeleteTablet for tablet b65132251435417ca88a258b567e0cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:05 UTC) from {username='oryx', principal='oryx/127.21.140.125@KRBTEST.COM'} at 127.0.0.1:34550
I20260928 21:07:05.162933 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28935
I20260928 21:07:05.164618 29560 tablet_replica.cc:330] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: stopping tablet replica
I20260928 21:07:05.164767 29560 raft_consensus.cc:2229] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:05.164919 29560 raft_consensus.cc:2258] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:05.165191 29558 ts_tablet_manager.cc:1905] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:05.165548 29560 ts_tablet_manager.cc:1905] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:05.167747 29560 ts_tablet_manager.cc:1918] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:05.167845 29560 log.cc:1196] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/b65132251435417ca88a258b567e0cb2
I20260928 21:07:05.168108 29560 ts_tablet_manager.cc:1939] T b65132251435417ca88a258b567e0cb2 P 35838c8b76f145188b251f569c192617: Deleting consensus metadata
I20260928 21:07:05.168759 29558 ts_tablet_manager.cc:1918] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:05.168853 29558 log.cc:1196] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/b65132251435417ca88a258b567e0cb2
I20260928 21:07:05.168840 28777 catalog_manager.cc:4841] TS 35838c8b76f145188b251f569c192617 (127.21.140.67:35435): tablet b65132251435417ca88a258b567e0cb2 (table test-table [id=cd7e917c9cae49d68a516f80ec193fb4]) successfully deleted
I20260928 21:07:05.169183 29558 ts_tablet_manager.cc:1939] T b65132251435417ca88a258b567e0cb2 P aafc3b4ed0bb4412ac5d83d5724f6229: Deleting consensus metadata
I20260928 21:07:05.170037 28774 catalog_manager.cc:4841] TS aafc3b4ed0bb4412ac5d83d5724f6229 (127.21.140.66:32849): tablet b65132251435417ca88a258b567e0cb2 (table test-table [id=cd7e917c9cae49d68a516f80ec193fb4]) successfully deleted
I20260928 21:07:05.173930 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 29163
I20260928 21:07:05.174100 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:05.173697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:40035 (local address 127.0.0.1:35908)
0928 21:07:05.173897 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:05.173997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:40035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 21:07:05.175835 28776 catalog_manager.cc:4584] TS 14a3755777754de48df817a03bf8a6d9 (127.21.140.65:40035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b65132251435417ca88a258b567e0cb2: Network error: Client connection negotiation failed: client connection to 127.21.140.65:40035: connect: Connection refused (error 111)
I20260928 21:07:05.180677 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 29299
I20260928 21:07:05.186386 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28651
I20260928 21:07:05.192371 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 28755
2026-09-28T21:07:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7662 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:07:05 dist-test-slave-d5wc krb5kdc[29563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:07:05 dist-test-slave-d5wc krb5kdc[29563](info): set up 2 sockets
Sep 28 21:07:05 dist-test-slave-d5wc krb5kdc[29563](info): commencing operation
krb5kdc: starting...
W20260928 21:07:06.276392 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:07:06 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab.
Entry for principal oryx/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab.
I20260928 21:07:06.407399 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44773
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab
--principal=oryx/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:06.521423 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:06.521685 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:06.521724 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:06.525804 29576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:06.525866 29576 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:07:06.525882 29576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:06.525900 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:06.525916 29576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:06.525931 29576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:06.531327 29576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.126.keytab
--principal=oryx/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:06.532287 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:06.533294 29576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:06.541945 29582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:06.542097 29581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:06.542119 29584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:06.542326 29576 server_base.cc:1034] running on GCE node
I20260928 21:07:06.542902 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:06.544075 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:06.545256 29576 hybrid_clock.cc:648] HybridClock initialized: now 1790629626545229 us; error 41 us; skew 500 ppm
Sep 28 21:07:06 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629626, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:06.548044 29576 init.cc:377] Logged in from keytab as oryx/127.21.140.126@KRBTEST.COM (short username oryx)
I20260928 21:07:06.548897 29576 webserver.cc:460] Webserver started at http://127.21.140.126:43899/ using document root <none> and password file <none>
I20260928 21:07:06.549464 29576 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:06.549551 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:06.549778 29576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:06.551807 29576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "cb6e153538504971bf53f45ac933a939"
format_stamp: "Formatted at 2026-09-28 21:07:06 on dist-test-slave-d5wc"
server_key: "402bb48476ae459cb97be061cea1b3af"
server_key_iv: "463dbf2c92fe8678bb15b3ec1fa5626d"
server_key_version: "encryptionkey@0"
I20260928 21:07:06.552362 29576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "cb6e153538504971bf53f45ac933a939"
format_stamp: "Formatted at 2026-09-28 21:07:06 on dist-test-slave-d5wc"
server_key: "402bb48476ae459cb97be061cea1b3af"
server_key_iv: "463dbf2c92fe8678bb15b3ec1fa5626d"
server_key_version: "encryptionkey@0"
I20260928 21:07:06.556010 29576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:07:06.558451 29591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:06.559664 29576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:07:06.559818 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "cb6e153538504971bf53f45ac933a939"
format_stamp: "Formatted at 2026-09-28 21:07:06 on dist-test-slave-d5wc"
server_key: "402bb48476ae459cb97be061cea1b3af"
server_key_iv: "463dbf2c92fe8678bb15b3ec1fa5626d"
server_key_version: "encryptionkey@0"
I20260928 21:07:06.559937 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:06.570346 29576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:06.572994 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:06.573194 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:06.580678 29576 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44773
I20260928 21:07:06.580688 29654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44773 every 8 connection(s)
I20260928 21:07:06.581619 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:07:06.584084 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 29576
I20260928 21:07:06.584218 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:07:06.584523 22065 external_mini_cluster.cc:1428] Setting key 6a019eae5c846fb69351ca4be48b9985
I20260928 21:07:06.584596 29655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:06.591988 29655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Bootstrap starting.
Sep 28 21:07:06 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:06.594599 29655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:06.595474 29655 log.cc:824] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:06.597440 29655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: No bootstrap required, opened a new log
I20260928 21:07:06.600399 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:06.586049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.21.140.126:44773)
0928 21:07:06.586475 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:06.586496 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:06.586538 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:07:06.587477 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:06.587500 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:06.587914 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:06.588303 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:06.588974 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.590439 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.591301 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.591963 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.592127 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:06.594924 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:06.594929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:06.594941 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:06.594969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:06.597205 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:06.597688 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:06.597694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:06.597700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:06.597848 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:06.598205 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:06.598208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:06.598210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:06.598712 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:06.598898 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:06.599183 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 21:07:06.599471 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260928 21:07:06.601491 29655 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e153538504971bf53f45ac933a939" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44773 } }
I20260928 21:07:06.601796 29655 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:06.601858 29655 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e153538504971bf53f45ac933a939, State: Initialized, Role: FOLLOWER
I20260928 21:07:06.602365 29655 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e153538504971bf53f45ac933a939" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44773 } }
I20260928 21:07:06.602515 29655 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:07:06.602591 29655 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:07:06.602695 29655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:06.603596 29655 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e153538504971bf53f45ac933a939" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44773 } }
I20260928 21:07:06.603940 29655 leader_election.cc:302] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6e153538504971bf53f45ac933a939; no voters:
I20260928 21:07:06.604257 29655 leader_election.cc:288] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:07:06.604372 29660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:06.604621 29660 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [term 1 LEADER]: Becoming Leader. State: Replica: cb6e153538504971bf53f45ac933a939, State: Running, Role: LEADER
I20260928 21:07:06.604897 29660 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e153538504971bf53f45ac933a939" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44773 } }
I20260928 21:07:06.605495 29655 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:06.606668 29661 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb6e153538504971bf53f45ac933a939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e153538504971bf53f45ac933a939" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44773 } } }
I20260928 21:07:06.606801 29661 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:06.607184 29669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:06.607574 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6e153538504971bf53f45ac933a939. Latest consensus state: current_term: 1 leader_uuid: "cb6e153538504971bf53f45ac933a939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e153538504971bf53f45ac933a939" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44773 } } }
I20260928 21:07:06.607673 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:06.610476 29669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:06.615863 29669 catalog_manager.cc:1347] Generated new cluster ID: 6db6e2bb26db42aeb18183a2ca984c5b
I20260928 21:07:06.615935 29669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:06.625036 29669 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:07:06.626225 29669 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:06.636257 29669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Generated new TSK 0
I20260928 21:07:06.637082 29669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab.
Entry for principal oryx/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab.
I20260928 21:07:06.672996 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44773
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:06.784385 29681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:06.784633 29681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:06.784677 29681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:06.788673 29681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:06.788723 29681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:06.788784 29681 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:07:06.788851 29681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:06.793900 29681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:44773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:06.794895 29681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:06.795843 29681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:06.805301 29689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:06.805383 29686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:06.805617 29681 server_base.cc:1034] running on GCE node
W20260928 21:07:06.805387 29687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:06.806185 29681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:06.806902 29681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:06.808103 29681 hybrid_clock.cc:648] HybridClock initialized: now 1790629626808092 us; error 27 us; skew 500 ppm
Sep 28 21:07:06 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629626, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:06.810978 29681 init.cc:377] Logged in from keytab as oryx/127.21.140.65@KRBTEST.COM (short username oryx)
I20260928 21:07:06.811851 29681 webserver.cc:460] Webserver started at http://127.21.140.65:45797/ using document root <none> and password file <none>
I20260928 21:07:06.812391 29681 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:06.812500 29681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:06.812728 29681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:06.814752 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "05b5f8728b794709b9e790d3257c3598"
format_stamp: "Formatted at 2026-09-28 21:07:06 on dist-test-slave-d5wc"
server_key: "c2ff9e3e1eadd3b300869fd2d9a6158d"
server_key_iv: "345cd6ca98bd4060863a2af8bc1dbee5"
server_key_version: "encryptionkey@0"
I20260928 21:07:06.815294 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "05b5f8728b794709b9e790d3257c3598"
format_stamp: "Formatted at 2026-09-28 21:07:06 on dist-test-slave-d5wc"
server_key: "c2ff9e3e1eadd3b300869fd2d9a6158d"
server_key_iv: "345cd6ca98bd4060863a2af8bc1dbee5"
server_key_version: "encryptionkey@0"
I20260928 21:07:06.818998 29681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:07:06.821455 29696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:06.822748 29681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:07:06.822888 29681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "05b5f8728b794709b9e790d3257c3598"
format_stamp: "Formatted at 2026-09-28 21:07:06 on dist-test-slave-d5wc"
server_key: "c2ff9e3e1eadd3b300869fd2d9a6158d"
server_key_iv: "345cd6ca98bd4060863a2af8bc1dbee5"
server_key_version: "encryptionkey@0"
I20260928 21:07:06.823009 29681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:06.843133 29681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:06.846764 29681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:06.847052 29681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:06.848690 29681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:06.848853 29681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:06.848978 29681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:06.849017 29681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:07:06 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629626, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:06.860220 29681 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:36193
I20260928 21:07:06.860294 29813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:36193 every 8 connection(s)
I20260928 21:07:06.861361 29681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:06.863317 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:06.849506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:43153 (local address 127.21.140.126:44773)
0928 21:07:06.849672 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:06.849677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:06.850684 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 21:07:06.852001 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:06.852005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:06.852071 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:06.852182 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:06.853843 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.854457 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.855638 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.855876 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.856021 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:06.859196 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:06.859199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:06.859203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:06.859238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:06.861025 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:06.861582 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:06.861587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:06.861588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:06.861681 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:06.862174 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:06.862177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:06.862179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:06.862409 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:06.862491 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:06.863057 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 21:07:06.863195 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:07:06.864064 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:06.849928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.21.140.65:43153)
0928 21:07:06.850486 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:06.850550 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:06.851691 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:06.852559 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:06.852570 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:06.853001 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:06.853641 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:06.853661 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.854596 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.854598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:06.855501 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:06.855510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.856069 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.856073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:06.856259 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:06.856884 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:06.856907 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:06.859010 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:06.861155 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:06.861159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:06.861170 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:06.861462 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:06.861786 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:06.861790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:06.861795 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:06.862055 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:06.862488 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:06.862493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:06.862713 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 21:07:06.862862 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 21:07:06.863164 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":113,"threads_started":1}
I20260928 21:07:06.865511 29814 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44773
I20260928 21:07:06.865667 29814 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:06.866235 29814 heartbeater.cc:511] Master 127.21.140.126:44773 requested a full tablet report, sending...
I20260928 21:07:06.867826 29607 ts_manager.cc:194] Registered new tserver with Master: 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193)
I20260928 21:07:06.869202 29607 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:43153
I20260928 21:07:06.869565 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 29681
I20260928 21:07:06.869719 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:07:06.870101 22065 external_mini_cluster.cc:1428] Setting key e8d5b4143487f9992aacb5f8f38c3fa7
I20260928 21:07:06.875494 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:06.867799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:48865 (local address 127.21.140.126:44773)
0928 21:07:06.867931 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:06.867935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:06.868122 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:07:06.868212 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:06.868214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:06.868253 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:06.868316 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:06.868810 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.869371 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.870627 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.870934 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.871056 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:06.872195 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:06.872201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:06.872208 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:06.872238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:06.873782 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:06.874371 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:06.874373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:06.874374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:06.874444 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:06.874760 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:06.874762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:06.874763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:06.874915 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:06.874990 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:06.875276 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 21:07:06.875369 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:07:06.875699 29820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:06.867737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.21.140.65:48865)
0928 21:07:06.868018 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:06.868034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:06.868133 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:06.868360 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:06.868363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:06.868576 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:06.868697 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:06.868704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.869508 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.869511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:06.870498 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:06.870509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:06.871074 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:06.871078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:06.871172 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:06.871707 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:06.871727 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:06.872055 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:06.873948 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:06.873953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:06.873956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:06.874269 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:06.874560 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:06.874566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:06.874570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:06.874668 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:06.875010 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:06.875016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:06.875125 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 21:07:06.875241 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 21:07:06.875401 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":8,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for oryx/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab.
Entry for principal oryx/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab.
I20260928 21:07:06.897198 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44773
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:07.008430 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:07.008706 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:07.008751 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:07.012912 29823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:07.012976 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:07.013023 29823 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:07:07.013057 29823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:07.018318 29823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:44773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:07.019320 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:07.020277 29823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:07.030149 29829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:07.030194 29831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:07.030195 29828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:07.030364 29823 server_base.cc:1034] running on GCE node
I20260928 21:07:07.030831 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:07.031471 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:07.032686 29823 hybrid_clock.cc:648] HybridClock initialized: now 1790629627032651 us; error 39 us; skew 500 ppm
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629627, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:07.035377 29823 init.cc:377] Logged in from keytab as oryx/127.21.140.66@KRBTEST.COM (short username oryx)
I20260928 21:07:07.036250 29823 webserver.cc:460] Webserver started at http://127.21.140.66:38351/ using document root <none> and password file <none>
I20260928 21:07:07.036814 29823 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:07.036903 29823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:07.037141 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:07.039264 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "f8cda18f67404197940f9d7276f234ac"
format_stamp: "Formatted at 2026-09-28 21:07:07 on dist-test-slave-d5wc"
server_key: "3fac7cc97f410d41284b35c0e3a33966"
server_key_iv: "b575b04f9e71c72e1245e7efc24181c1"
server_key_version: "encryptionkey@0"
I20260928 21:07:07.039841 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "f8cda18f67404197940f9d7276f234ac"
format_stamp: "Formatted at 2026-09-28 21:07:07 on dist-test-slave-d5wc"
server_key: "3fac7cc97f410d41284b35c0e3a33966"
server_key_iv: "b575b04f9e71c72e1245e7efc24181c1"
server_key_version: "encryptionkey@0"
I20260928 21:07:07.043658 29823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:07:07.046198 29838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:07.047514 29823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:07:07.047647 29823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "f8cda18f67404197940f9d7276f234ac"
format_stamp: "Formatted at 2026-09-28 21:07:07 on dist-test-slave-d5wc"
server_key: "3fac7cc97f410d41284b35c0e3a33966"
server_key_iv: "b575b04f9e71c72e1245e7efc24181c1"
server_key_version: "encryptionkey@0"
I20260928 21:07:07.047763 29823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:07.070120 29823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:07.073313 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:07.073515 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:07.074985 29823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:07.075065 29823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:07.075157 29823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:07.075194 29823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629627, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:07.085780 29823 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:40589
I20260928 21:07:07.085839 29954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:40589 every 8 connection(s)
I20260928 21:07:07.086958 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:07.088932 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.075909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:56189 (local address 127.21.140.126:44773)
0928 21:07:07.076084 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.076091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.076896 (+ 805us) server_negotiation.cc:407] Connection header received
0928 21:07:07.078069 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.078074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.078138 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.078248 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.079538 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.080069 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.081375 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.081661 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.081780 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.084812 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:07.084816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:07.084818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:07.084846 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:07.086721 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.087397 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.087401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.087403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.087505 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.087896 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.087901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.087903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.088103 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:07.088183 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.088663 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.088768 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:07:07.089624 29847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.076291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.21.140.66:56189)
0928 21:07:07.076731 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.076781 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.077796 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.078444 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.078453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.078882 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:07.079381 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.079396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.080201 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.080207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.081233 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.081243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.081831 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.081836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.082057 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.082660 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:07.082676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:07.084635 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:07.086850 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.086854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.086926 (+ 72us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.087254 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.087598 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.087602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.087604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.087764 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.088198 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:07.088203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:07.088432 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.088589 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.088824 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":134,"threads_started":1}
I20260928 21:07:07.091038 29955 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44773
I20260928 21:07:07.091236 29955 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:07.091773 29955 heartbeater.cc:511] Master 127.21.140.126:44773 requested a full tablet report, sending...
I20260928 21:07:07.092872 29608 ts_manager.cc:194] Registered new tserver with Master: f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589)
I20260928 21:07:07.093748 29608 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:56189
I20260928 21:07:07.093864 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 29823
I20260928 21:07:07.094065 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:07:07.094396 22065 external_mini_cluster.cc:1428] Setting key 158656e3556b276b02611feac989134c
I20260928 21:07:07.101025 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.093027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.21.140.66:54527)
0928 21:07:07.093277 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.093290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.093368 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.093679 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.093684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.093909 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:07.094075 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.094085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.095146 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.095149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.096059 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.096068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.096739 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.096741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.096811 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.097298 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:07.097309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:07.097618 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:07.099366 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.099369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.099372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.099667 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.099993 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.099995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.099997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.100071 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.100591 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:07.100594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:07.100663 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.100742 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.100848 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260928 21:07:07.101214 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.093089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:54527 (local address 127.21.140.126:44773)
0928 21:07:07.093214 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.093219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.093360 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:07:07.093471 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.093475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.093541 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.093617 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.094273 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.095020 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.096197 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.096513 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.096591 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.097755 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:07.097759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:07.097762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:07.097784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:07.099236 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.099781 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.099787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.099790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.099886 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.100167 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.100171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.100173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.100490 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:07.100608 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.100843 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.101015 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for oryx/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab.
Entry for principal oryx/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab.
I20260928 21:07:07.122771 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44773
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:07.232900 29964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:07.233156 29964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:07.233201 29964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:07.237257 29964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:07.237313 29964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:07.237353 29964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:07:07.237426 29964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:07.242800 29964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.140.126:44773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:07.243865 29964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:07.244875 29964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:07.254832 29970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:07.254824 29969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:07.254824 29972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:07.254901 29964 server_base.cc:1034] running on GCE node
I20260928 21:07:07.255537 29964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:07.256204 29964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:07.257372 29964 hybrid_clock.cc:648] HybridClock initialized: now 1790629627257338 us; error 50 us; skew 500 ppm
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629627, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:07.260381 29964 init.cc:377] Logged in from keytab as oryx/127.21.140.67@KRBTEST.COM (short username oryx)
I20260928 21:07:07.261332 29964 webserver.cc:460] Webserver started at http://127.21.140.67:37479/ using document root <none> and password file <none>
I20260928 21:07:07.261897 29964 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:07.262022 29964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:07.262239 29964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:07.264519 29964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "a5625c6033eb47aa91f1825dc77c7d64"
format_stamp: "Formatted at 2026-09-28 21:07:07 on dist-test-slave-d5wc"
server_key: "c9d093047da8fa2c0a7449f491e60b8b"
server_key_iv: "45f5d549f51635075737cd1964b22f12"
server_key_version: "encryptionkey@0"
I20260928 21:07:07.265184 29964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "a5625c6033eb47aa91f1825dc77c7d64"
format_stamp: "Formatted at 2026-09-28 21:07:07 on dist-test-slave-d5wc"
server_key: "c9d093047da8fa2c0a7449f491e60b8b"
server_key_iv: "45f5d549f51635075737cd1964b22f12"
server_key_version: "encryptionkey@0"
I20260928 21:07:07.269089 29964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:07:07.271627 29979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:07.273051 29964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:07:07.273188 29964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "a5625c6033eb47aa91f1825dc77c7d64"
format_stamp: "Formatted at 2026-09-28 21:07:07 on dist-test-slave-d5wc"
server_key: "c9d093047da8fa2c0a7449f491e60b8b"
server_key_iv: "45f5d549f51635075737cd1964b22f12"
server_key_version: "encryptionkey@0"
I20260928 21:07:07.273305 29964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:07.290917 29964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:07.294519 29964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:07.294732 29964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:07.296159 29964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:07.296209 29964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:07.296248 29964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:07.296305 29964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629627, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:07.307062 29964 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:38183
I20260928 21:07:07.307147 30095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:38183 every 8 connection(s)
I20260928 21:07:07.308164 29964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:07:07.309073 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 29964
I20260928 21:07:07.309217 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:07:07.309544 22065 external_mini_cluster.cc:1428] Setting key e3fab92e5782d006205e63debbcc21a1
I20260928 21:07:07.311476 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.297349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:46901 (local address 127.21.140.126:44773)
0928 21:07:07.297506 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.297511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.298525 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 21:07:07.299879 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.299883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.299933 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.300015 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.301721 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.302496 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.303767 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.304016 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.304143 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.307276 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:07.307280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:07.307283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:07.307311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:07.309180 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.309720 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.309724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.309726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.309818 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.310286 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.310290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.310292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.310506 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:07.310633 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.311166 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.311296 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:07:07.312316 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.297777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.21.140.67:46901)
0928 21:07:07.298327 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.298392 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.299628 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.300228 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.300239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.300787 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:07.301542 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.301566 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.302646 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.302650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.303611 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.303621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.304183 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.304187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.304391 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.305025 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:07.305049 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:07.307101 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:07.309312 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.309319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.309332 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.309627 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.309949 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.309953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.309956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.310161 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.310630 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:07.310636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:07.310889 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.311101 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.311442 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":187,"threads_started":1}
I20260928 21:07:07.313566 30096 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44773
I20260928 21:07:07.313717 30096 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:07.314253 30096 heartbeater.cc:511] Master 127.21.140.126:44773 requested a full tablet report, sending...
I20260928 21:07:07.315260 29607 ts_manager.cc:194] Registered new tserver with Master: a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:07.316032 29607 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:46901
I20260928 21:07:07.322659 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.315738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.21.140.67:41057)
0928 21:07:07.316120 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.316136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.316246 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.316491 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.316494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.316659 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:07.316749 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.316754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.317505 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.317509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.318136 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.318144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.318594 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.318597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.318688 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.319154 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:07.319166 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:07.319505 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:07.321031 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.321037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.321040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.321231 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.321537 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.321540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.321542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.321631 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.322054 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:07.322056 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:07.322178 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.322326 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.322476 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20260928 21:07:07.322731 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.315841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:41057 (local address 127.21.140.126:44773)
0928 21:07:07.315965 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.315969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.316209 (+ 240us) server_negotiation.cc:407] Connection header received
0928 21:07:07.316335 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.316338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.316385 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.316468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.316866 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.317371 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.318249 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.318484 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.318592 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.319664 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:07.319668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:07.319670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:07.319694 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:07.320913 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.321336 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.321341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.321343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.321436 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.321726 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.321731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.321732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.321915 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:07.322063 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.322416 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.322572 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:07:07.323436 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:07:07.348104 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:07.341361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.21.140.126:44773)
0928 21:07:07.341527 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.341530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.341542 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:07:07.341581 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.341583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.341620 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.341718 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.342493 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.343034 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.343894 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.344116 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.344234 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.347925 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41934: BlockingRecv error: recv got EOF from 127.0.0.1:41934 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:07.358126 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.349508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.21.140.126:44773)
0928 21:07:07.349657 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.349661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.349719 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:07:07.349867 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.349870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.349953 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.350123 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.350780 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.351475 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.352440 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.352662 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.352751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.355176 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:07.355179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:07.355181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:07.355205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:07.356636 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.357074 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.357076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.357078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.357154 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.357387 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.357389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.357390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.357548 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:07.357646 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.357838 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.357929 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 21:07:07.360847 29608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:07:07.364152 29608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:07.380548 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.373560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38183 (local address 127.0.0.1:43642)
0928 21:07:07.374851 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.374861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.374964 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.375515 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.375518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.375538 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:07.375688 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.375696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.377043 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.377046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.378550 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.378562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.379892 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.379896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.380080 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.380138 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.380248 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.380377 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":57,"thread_start_us":88,"threads_started":1}
I20260928 21:07:07.381049 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.374028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.21.140.67:38183)
0928 21:07:07.374527 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.374540 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.374953 (+ 413us) server_negotiation.cc:407] Connection header received
0928 21:07:07.375127 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.375134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.375336 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.375507 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:07.375830 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.376915 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.378704 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.379534 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.379626 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.379815 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.380777 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.380910 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":113,"threads_started":1}
I20260928 21:07:07.382052 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.373275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36193 (local address 127.0.0.1:45950)
0928 21:07:07.374610 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.374622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.374712 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.375177 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.375184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.375208 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:07.375393 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.375406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.377889 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.377893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.379395 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.379405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.381589 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.381595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.381709 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.381732 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.381813 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.381891 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":6,"thread_start_us":141,"threads_started":1}
I20260928 21:07:07.382526 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.374028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.21.140.65:36193)
0928 21:07:07.374448 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.374456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.374728 (+ 272us) server_negotiation.cc:407] Connection header received
0928 21:07:07.374819 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.374824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.375020 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.375177 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:07.375518 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.377752 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.379548 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.381237 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.381352 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.381514 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.382252 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.382392 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":102,"threads_started":1}
I20260928 21:07:07.382833 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.373211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:40589 (local address 127.0.0.1:45754)
0928 21:07:07.374152 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.374209 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.374337 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.375532 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.375535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.375552 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:07.375690 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.375701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.377552 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.377559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.380066 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.380109 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.381851 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.381855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.382034 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.382056 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.382448 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.382703 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":217,"thread_start_us":96,"threads_started":1}
I20260928 21:07:07.383244 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.374051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.21.140.66:40589)
0928 21:07:07.374703 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.374714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.374732 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:07:07.374808 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.374813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.375020 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.375177 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:07.375846 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.377389 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.380332 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.381714 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.381851 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.382070 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.382408 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.383099 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":107,"threads_started":1}
I20260928 21:07:07.386636 30029 tablet_service.cc:1467] Processing CreateTablet for tablet c76a2ec895ed46bc8518bb117349a95f (DEFAULT_TABLE table=test-table [id=ded3bd01e0974cc9aeb48f0a724461f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:07.386615 29888 tablet_service.cc:1467] Processing CreateTablet for tablet c76a2ec895ed46bc8518bb117349a95f (DEFAULT_TABLE table=test-table [id=ded3bd01e0974cc9aeb48f0a724461f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:07.387770 29888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76a2ec895ed46bc8518bb117349a95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.387811 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76a2ec895ed46bc8518bb117349a95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.392481 29746 tablet_service.cc:1467] Processing CreateTablet for tablet c76a2ec895ed46bc8518bb117349a95f (DEFAULT_TABLE table=test-table [id=ded3bd01e0974cc9aeb48f0a724461f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:07.393631 29746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76a2ec895ed46bc8518bb117349a95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.396245 30125 tablet_bootstrap.cc:492] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Bootstrap starting.
I20260928 21:07:07.396685 30124 tablet_bootstrap.cc:492] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Bootstrap starting.
I20260928 21:07:07.398610 30125 tablet_bootstrap.cc:654] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.398810 30124 tablet_bootstrap.cc:654] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.399541 30124 log.cc:824] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:07.399541 30125 log.cc:824] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:07.401486 30128 tablet_bootstrap.cc:492] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Bootstrap starting.
I20260928 21:07:07.401782 30125 tablet_bootstrap.cc:492] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: No bootstrap required, opened a new log
I20260928 21:07:07.401818 30124 tablet_bootstrap.cc:492] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: No bootstrap required, opened a new log
I20260928 21:07:07.401942 30125 ts_tablet_manager.cc:1397] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:07:07.402060 30124 ts_tablet_manager.cc:1397] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:07:07.404163 30128 tablet_bootstrap.cc:654] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.405122 30128 log.cc:824] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:07.405098 30125 raft_consensus.cc:348] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.405434 30125 raft_consensus.cc:374] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.405500 30125 raft_consensus.cc:729] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cda18f67404197940f9d7276f234ac, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.405942 30125 consensus_queue.cc:260] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.406543 30124 raft_consensus.cc:348] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.406880 30124 raft_consensus.cc:374] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.406940 30124 raft_consensus.cc:729] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5625c6033eb47aa91f1825dc77c7d64, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.407127 30125 ts_tablet_manager.cc:1428] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260928 21:07:07.407418 30124 consensus_queue.cc:260] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.407780 29955 heartbeater.cc:503] Master 127.21.140.126:44773 was elected leader, sending a full tablet report...
I20260928 21:07:07.407989 30128 tablet_bootstrap.cc:492] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: No bootstrap required, opened a new log
I20260928 21:07:07.408185 30128 ts_tablet_manager.cc:1397] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:07:07.408519 30124 ts_tablet_manager.cc:1428] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:07:07.408946 30096 heartbeater.cc:503] Master 127.21.140.126:44773 was elected leader, sending a full tablet report...
I20260928 21:07:07.412962 30128 raft_consensus.cc:348] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.413219 30128 raft_consensus.cc:374] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.413296 30128 raft_consensus.cc:729] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.413764 30128 consensus_queue.cc:260] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.414436 29814 heartbeater.cc:503] Master 127.21.140.126:44773 was elected leader, sending a full tablet report...
I20260928 21:07:07.414783 30128 ts_tablet_manager.cc:1428] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260928 21:07:07.470278 30132 raft_consensus.cc:482] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:07.470523 30132 raft_consensus.cc:504] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.471747 30132 leader_election.cc:288] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:07.476186 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.472134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:41723 (local address 127.21.140.67:38183)
0928 21:07:07.472266 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.472273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.472532 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:07:07.472602 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.472606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.472670 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.472750 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:07.473060 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.473651 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.475218 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.475615 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.475707 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.475785 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.475962 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.476050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:07:07.476224 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.472162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:59865 (local address 127.21.140.66:40589)
0928 21:07:07.472283 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.472299 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.472702 (+ 403us) server_negotiation.cc:407] Connection header received
0928 21:07:07.472795 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.472797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.472842 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.472923 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:07.473255 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.473898 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.475262 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.475647 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.475783 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.475834 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.476038 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.476119 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:07:07.476235 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.472004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38183 (local address 127.21.140.65:41723)
0928 21:07:07.472394 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.472406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.472512 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.472776 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.472778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.472795 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:07.472937 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.472943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.473861 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.473864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.475105 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.475114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.475739 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.475745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.475833 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.475864 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.475936 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.476028 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":20,"thread_start_us":146,"threads_started":1}
I20260928 21:07:07.476635 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.472018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:40589 (local address 127.21.140.65:59865)
0928 21:07:07.472588 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.472602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.472708 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.472940 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.472942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.472956 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:07.473116 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.473124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.474039 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.474042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.475117 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.475126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.475792 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.475798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.475892 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.475933 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.476009 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.476070 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":18,"thread_start_us":121,"threads_started":1}
I20260928 21:07:07.476868 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a2ec895ed46bc8518bb117349a95f" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64" is_pre_election: true
I20260928 21:07:07.477193 30049 raft_consensus.cc:2454] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 0.
I20260928 21:07:07.477304 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a2ec895ed46bc8518bb117349a95f" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac" is_pre_election: true
I20260928 21:07:07.477625 29909 raft_consensus.cc:2454] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 0.
I20260928 21:07:07.477670 29698 leader_election.cc:302] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, a5625c6033eb47aa91f1825dc77c7d64; no voters:
I20260928 21:07:07.478019 30132 raft_consensus.cc:2781] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:07.478108 30132 raft_consensus.cc:482] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:07.478176 30132 raft_consensus.cc:3037] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:07.479192 30132 raft_consensus.cc:504] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.479532 30132 leader_election.cc:288] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 election: Requested vote from peers f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:07.479827 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a2ec895ed46bc8518bb117349a95f" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac"
I20260928 21:07:07.479879 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76a2ec895ed46bc8518bb117349a95f" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64"
I20260928 21:07:07.479950 29909 raft_consensus.cc:3037] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:07.479980 30049 raft_consensus.cc:3037] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:07.481247 29909 raft_consensus.cc:2454] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 1.
I20260928 21:07:07.481247 30049 raft_consensus.cc:2454] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 1.
I20260928 21:07:07.481535 29698 leader_election.cc:302] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, a5625c6033eb47aa91f1825dc77c7d64; no voters:
I20260928 21:07:07.481705 30132 raft_consensus.cc:2781] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:07.481925 30132 raft_consensus.cc:686] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 1 LEADER]: Becoming Leader. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Running, Role: LEADER
I20260928 21:07:07.482301 30132 consensus_queue.cc:237] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.485325 29607 catalog_manager.cc:5495] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b5f8728b794709b9e790d3257c3598 (127.21.140.65). New cstate: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } health_report { overall_health: UNKNOWN } } }
I20260928 21:07:07.506336 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.502782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.21.140.65:36193)
0928 21:07:07.502917 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.502920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.502932 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:07:07.503001 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.503004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.503062 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.503148 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:07.503402 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.503926 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.504754 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.505034 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.505133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.505398 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:07.505992 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:07.506108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.506174 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.506226 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:07:07.512790 29909 raft_consensus.cc:1261] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Refusing update from remote peer 05b5f8728b794709b9e790d3257c3598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:07.512790 30049 raft_consensus.cc:1261] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Refusing update from remote peer 05b5f8728b794709b9e790d3257c3598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:07.513507 30135 consensus_queue.cc:1035] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:07.513705 30132 consensus_queue.cc:1035] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:07.520473 30138 mvcc.cc:205] Tried to move back new op lower bound from 7334418954283139072 to 7334418954170187776. Current Snapshot: MvccSnapshot[applied={T|T < 7334418954283139072}]
I20260928 21:07:07.522260 30139 mvcc.cc:205] Tried to move back new op lower bound from 7334418954283139072 to 7334418954170187776. Current Snapshot: MvccSnapshot[applied={T|T < 7334418954283139072}]
I20260928 21:07:07.526746 30140 mvcc.cc:205] Tried to move back new op lower bound from 7334418954283139072 to 7334418954170187776. Current Snapshot: MvccSnapshot[applied={T|T < 7334418954283139072 or (T in {7334418954283139072})}]
I20260928 21:07:07.529345 29608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:07.529561 29608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:07.531808 29608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Sending DeleteTablet for 3 replicas of tablet c76a2ec895ed46bc8518bb117349a95f
I20260928 21:07:07.532490 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet c76a2ec895ed46bc8518bb117349a95f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:07 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45754
I20260928 21:07:07.532629 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet c76a2ec895ed46bc8518bb117349a95f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:07 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45950
I20260928 21:07:07.532682 30029 tablet_service.cc:1514] Processing DeleteTablet for tablet c76a2ec895ed46bc8518bb117349a95f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:07 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:43642
I20260928 21:07:07.533126 30147 tablet_replica.cc:330] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: stopping tablet replica
I20260928 21:07:07.533361 30149 tablet_replica.cc:330] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: stopping tablet replica
I20260928 21:07:07.533432 30147 raft_consensus.cc:2229] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:07.533735 30149 raft_consensus.cc:2229] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:07.533742 30147 raft_consensus.cc:2258] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:07.533149 30148 tablet_replica.cc:330] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: stopping tablet replica
I20260928 21:07:07.533864 29608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:07:07.534300 29608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:07.534300 30148 raft_consensus.cc:2229] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:07:07.534693 30149 raft_consensus.cc:2258] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:07.535125 30148 raft_consensus.cc:2258] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:07.535292 30147 ts_tablet_manager.cc:1905] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:07.536617 30148 ts_tablet_manager.cc:1905] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:07.538275 30147 ts_tablet_manager.cc:1918] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:07.538376 30147 log.cc:1196] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/c76a2ec895ed46bc8518bb117349a95f
I20260928 21:07:07.538726 30147 ts_tablet_manager.cc:1939] T c76a2ec895ed46bc8518bb117349a95f P f8cda18f67404197940f9d7276f234ac: Deleting consensus metadata
I20260928 21:07:07.539407 30029 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca5737e4006499cba87710a26cc71f3 (DEFAULT_TABLE table=test-table [id=cd834cb7a20f494d9dd1b5f7cdbfc8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:07.539676 30148 ts_tablet_manager.cc:1918] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:07.539703 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca5737e4006499cba87710a26cc71f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.539834 30148 log.cc:1196] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/c76a2ec895ed46bc8518bb117349a95f
I20260928 21:07:07.540167 30148 ts_tablet_manager.cc:1939] T c76a2ec895ed46bc8518bb117349a95f P 05b5f8728b794709b9e790d3257c3598: Deleting consensus metadata
I20260928 21:07:07.541926 30124 tablet_bootstrap.cc:492] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Bootstrap starting.
I20260928 21:07:07.542294 29592 catalog_manager.cc:4841] TS 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193): tablet c76a2ec895ed46bc8518bb117349a95f (table test-table [id=ded3bd01e0974cc9aeb48f0a724461f7]) successfully deleted
I20260928 21:07:07.542830 29746 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca5737e4006499cba87710a26cc71f3 (DEFAULT_TABLE table=test-table [id=cd834cb7a20f494d9dd1b5f7cdbfc8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:07.542950 30124 tablet_bootstrap.cc:654] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.542958 29594 catalog_manager.cc:4841] TS f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589): tablet c76a2ec895ed46bc8518bb117349a95f (table test-table [id=ded3bd01e0974cc9aeb48f0a724461f7]) successfully deleted
I20260928 21:07:07.543068 29746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca5737e4006499cba87710a26cc71f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.543417 29888 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca5737e4006499cba87710a26cc71f3 (DEFAULT_TABLE table=test-table [id=cd834cb7a20f494d9dd1b5f7cdbfc8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:07.543671 29888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca5737e4006499cba87710a26cc71f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.544078 30124 tablet_bootstrap.cc:492] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: No bootstrap required, opened a new log
I20260928 21:07:07.544145 30124 ts_tablet_manager.cc:1397] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:07:07.544549 30124 raft_consensus.cc:348] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.544648 30124 raft_consensus.cc:374] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.544678 30124 raft_consensus.cc:729] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5625c6033eb47aa91f1825dc77c7d64, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.544803 30124 consensus_queue.cc:260] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.545066 30124 ts_tablet_manager.cc:1428] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:07:07.545251 30128 tablet_bootstrap.cc:492] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Bootstrap starting.
I20260928 21:07:07.545815 30125 tablet_bootstrap.cc:492] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Bootstrap starting.
I20260928 21:07:07.546646 30128 tablet_bootstrap.cc:654] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.546723 30125 tablet_bootstrap.cc:654] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.547260 30149 ts_tablet_manager.cc:1905] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:07.547997 30125 tablet_bootstrap.cc:492] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: No bootstrap required, opened a new log
I20260928 21:07:07.548100 30125 ts_tablet_manager.cc:1397] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:07:07.548580 30125 raft_consensus.cc:348] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.548697 30125 raft_consensus.cc:374] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.548738 30125 raft_consensus.cc:729] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cda18f67404197940f9d7276f234ac, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.548868 30125 consensus_queue.cc:260] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.549171 30125 ts_tablet_manager.cc:1428] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:07.549434 30128 tablet_bootstrap.cc:492] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: No bootstrap required, opened a new log
I20260928 21:07:07.549518 30128 ts_tablet_manager.cc:1397] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:07:07.549664 30149 ts_tablet_manager.cc:1918] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:07.549741 30149 log.cc:1196] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/c76a2ec895ed46bc8518bb117349a95f
I20260928 21:07:07.550060 30149 ts_tablet_manager.cc:1939] T c76a2ec895ed46bc8518bb117349a95f P a5625c6033eb47aa91f1825dc77c7d64: Deleting consensus metadata
I20260928 21:07:07.550089 30128 raft_consensus.cc:348] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.550206 30128 raft_consensus.cc:374] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.550239 30128 raft_consensus.cc:729] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.550348 30128 consensus_queue.cc:260] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.550573 30128 ts_tablet_manager.cc:1428] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:07.551354 29593 catalog_manager.cc:4841] TS a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183): tablet c76a2ec895ed46bc8518bb117349a95f (table test-table [id=ded3bd01e0974cc9aeb48f0a724461f7]) successfully deleted
W20260928 21:07:07.560283 30097 tablet.cc:2369] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:07:07.588438 29956 tablet.cc:2369] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:07:07.612890 29815 tablet.cc:2369] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:07:07.891839 30132 raft_consensus.cc:482] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:07.891984 30132 raft_consensus.cc:504] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.892357 30132 leader_election.cc:288] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:07.892848 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca5737e4006499cba87710a26cc71f3" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac" is_pre_election: true
I20260928 21:07:07.892851 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca5737e4006499cba87710a26cc71f3" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64" is_pre_election: true
I20260928 21:07:07.893019 29909 raft_consensus.cc:2454] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 0.
I20260928 21:07:07.893031 30049 raft_consensus.cc:2454] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 0.
I20260928 21:07:07.893370 29699 leader_election.cc:302] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:07.893563 30132 raft_consensus.cc:2781] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:07.893625 30132 raft_consensus.cc:482] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:07.893654 30132 raft_consensus.cc:3037] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:07.894744 30132 raft_consensus.cc:504] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.895125 30132 leader_election.cc:288] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 election: Requested vote from peers f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:07.895498 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca5737e4006499cba87710a26cc71f3" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac"
I20260928 21:07:07.895625 29909 raft_consensus.cc:3037] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:07.895615 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca5737e4006499cba87710a26cc71f3" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64"
I20260928 21:07:07.895741 30049 raft_consensus.cc:3037] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:07.896544 29909 raft_consensus.cc:2454] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 1.
I20260928 21:07:07.896716 30049 raft_consensus.cc:2454] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 1.
I20260928 21:07:07.896831 29699 leader_election.cc:302] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:07.897017 30132 raft_consensus.cc:2781] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:07.897105 30132 raft_consensus.cc:686] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 1 LEADER]: Becoming Leader. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Running, Role: LEADER
I20260928 21:07:07.897243 30132 consensus_queue.cc:237] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.898891 29608 catalog_manager.cc:5495] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b5f8728b794709b9e790d3257c3598 (127.21.140.65). New cstate: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:07:07 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629626, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.126@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:07.945286 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.935652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44773 (local address 127.0.0.1:41948)
0928 21:07:07.936062 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:07.936074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:07.936172 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:07.936413 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:07.936415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:07.936809 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:07.936946 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.936952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.937718 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.937721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.938401 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:07.938410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.938828 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.938831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:07.938927 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.939557 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:07.939568 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:07.940962 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:07.943279 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.943289 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.943295 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.943653 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.944033 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:07.944036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:07.944038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:07.944134 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:07.944646 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:07.944653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:07.944798 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 21:07:07.944918 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 21:07:07.945079 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":93,"threads_started":1}
I20260928 21:07:07.945286 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:07.935761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.21.140.126:44773)
0928 21:07:07.936037 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:07.936044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:07.936149 (+ 105us) server_negotiation.cc:407] Connection header received
0928 21:07:07.936241 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:07.936243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:07.936283 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:07.936381 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:07.937094 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.937622 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.938511 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:07.938732 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:07.938809 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:07.941095 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:07.941098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:07.941100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:07.941121 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:07.943173 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.943801 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.943805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.943807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.943923 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:07.944257 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:07.944260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:07.944262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:07.944528 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:07.944627 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:07.944963 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 21:07:07.945159 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":204,"thread_start_us":139,"threads_started":1}
I20260928 21:07:07.949220 29608 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:41948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:07:07.950229 29608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:07.955276 29746 tablet_service.cc:1467] Processing CreateTablet for tablet ba3fa0c19ef145a3b9264b2b2c7e72e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4e3aef09e9046a8be6dd30cd9520699]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:07:07.955475 30029 tablet_service.cc:1467] Processing CreateTablet for tablet ba3fa0c19ef145a3b9264b2b2c7e72e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4e3aef09e9046a8be6dd30cd9520699]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:07:07.955466 29888 tablet_service.cc:1467] Processing CreateTablet for tablet ba3fa0c19ef145a3b9264b2b2c7e72e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4e3aef09e9046a8be6dd30cd9520699]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:07:07.955724 29746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3fa0c19ef145a3b9264b2b2c7e72e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.955837 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3fa0c19ef145a3b9264b2b2c7e72e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.955837 29888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3fa0c19ef145a3b9264b2b2c7e72e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:07.958086 30128 tablet_bootstrap.cc:492] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: Bootstrap starting.
I20260928 21:07:07.958631 30125 tablet_bootstrap.cc:492] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: Bootstrap starting.
I20260928 21:07:07.958992 30128 tablet_bootstrap.cc:654] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.959564 30125 tablet_bootstrap.cc:654] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.960112 30124 tablet_bootstrap.cc:492] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: Bootstrap starting.
I20260928 21:07:07.960657 30125 tablet_bootstrap.cc:492] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: No bootstrap required, opened a new log
I20260928 21:07:07.960726 30125 ts_tablet_manager.cc:1397] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:07:07.960939 30124 tablet_bootstrap.cc:654] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:07.961195 30125 raft_consensus.cc:348] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.961328 30125 raft_consensus.cc:374] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.961369 30125 raft_consensus.cc:729] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cda18f67404197940f9d7276f234ac, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.961504 30125 consensus_queue.cc:260] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.961818 30125 ts_tablet_manager.cc:1428] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:07.962108 30124 tablet_bootstrap.cc:492] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: No bootstrap required, opened a new log
I20260928 21:07:07.962191 30128 tablet_bootstrap.cc:492] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: No bootstrap required, opened a new log
I20260928 21:07:07.962199 30124 ts_tablet_manager.cc:1397] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 21:07:07.962281 30128 ts_tablet_manager.cc:1397] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:07:07.962599 30124 raft_consensus.cc:348] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.962617 30128 raft_consensus.cc:348] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.962714 30124 raft_consensus.cc:374] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.962754 30124 raft_consensus.cc:729] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5625c6033eb47aa91f1825dc77c7d64, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.962744 30128 raft_consensus.cc:374] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:07.962710 30130 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:07.962790 30128 raft_consensus.cc:729] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Initialized, Role: FOLLOWER
I20260928 21:07:07.962839 30130 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:07.962867 30124 consensus_queue.cc:260] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.962917 30128 consensus_queue.cc:260] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:07.963137 30124 ts_tablet_manager.cc:1428] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:07.963191 30128 ts_tablet_manager.cc:1428] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:07.963451 30131 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:07.963608 30131 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:07.963781 30132 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:07.963907 30132 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:08.168363 30132 raft_consensus.cc:482] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:08.168514 30132 raft_consensus.cc:504] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.169087 30132 leader_election.cc:288] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:08.169538 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3fa0c19ef145a3b9264b2b2c7e72e2" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac" is_pre_election: true
I20260928 21:07:08.169700 29909 raft_consensus.cc:2454] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 0.
I20260928 21:07:08.169663 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3fa0c19ef145a3b9264b2b2c7e72e2" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64" is_pre_election: true
I20260928 21:07:08.169837 30049 raft_consensus.cc:2454] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 0.
I20260928 21:07:08.170042 29699 leader_election.cc:302] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:08.170245 30132 raft_consensus.cc:2781] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:08.170323 30132 raft_consensus.cc:482] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:08.170365 30132 raft_consensus.cc:3037] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.171320 30132 raft_consensus.cc:504] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.171669 30132 leader_election.cc:288] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 election: Requested vote from peers f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:08.172078 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3fa0c19ef145a3b9264b2b2c7e72e2" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64"
I20260928 21:07:08.172094 29909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3fa0c19ef145a3b9264b2b2c7e72e2" candidate_uuid: "05b5f8728b794709b9e790d3257c3598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac"
I20260928 21:07:08.172248 29909 raft_consensus.cc:3037] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.172253 30049 raft_consensus.cc:3037] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.173182 29909 raft_consensus.cc:2454] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 1.
I20260928 21:07:08.173223 30049 raft_consensus.cc:2454] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b5f8728b794709b9e790d3257c3598 in term 1.
I20260928 21:07:08.173569 29699 leader_election.cc:302] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, a5625c6033eb47aa91f1825dc77c7d64, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:08.173739 30132 raft_consensus.cc:2781] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:08.173834 30132 raft_consensus.cc:686] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [term 1 LEADER]: Becoming Leader. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Running, Role: LEADER
I20260928 21:07:08.174044 30132 consensus_queue.cc:237] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.174659 30135 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: TxnStatusTablet state changed. Reason: New leader 05b5f8728b794709b9e790d3257c3598. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.174773 30135 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:07:08.174990 30166 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:07:08.175149 30166 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:07:08.175769 29608 catalog_manager.cc:5495] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b5f8728b794709b9e790d3257c3598 (127.21.140.65). New cstate: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } health_report { overall_health: UNKNOWN } } }
I20260928 21:07:08.204517 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.200523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36193 (local address 127.0.0.1:45972)
0928 21:07:08.200700 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.200720 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.200873 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.201245 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.201249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.201257 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:07:08.201382 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.201387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.202292 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.202298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.203023 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.203187 (+ 164us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.203696 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.203699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.203806 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.203849 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.204253 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.204266 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.204325 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.204403 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 21:07:08.204556 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.200702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.21.140.65:36193)
0928 21:07:08.200992 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.200999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.201017 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:07:08.201071 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.201073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.201118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.201221 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:08.201508 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.202173 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.203330 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.203581 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.203694 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.204060 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.204153 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.204234 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.204357 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.204414 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260928 21:07:08.207528 30049 raft_consensus.cc:1261] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Refusing update from remote peer 05b5f8728b794709b9e790d3257c3598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.207599 29909 raft_consensus.cc:1261] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Refusing update from remote peer 05b5f8728b794709b9e790d3257c3598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.208053 30132 consensus_queue.cc:1035] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.208269 30135 consensus_queue.cc:1035] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.209640 30130 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: TxnStatusTablet state changed. Reason: New leader 05b5f8728b794709b9e790d3257c3598. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.209758 30130 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:08.210882 30131 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: TxnStatusTablet state changed. Reason: New leader 05b5f8728b794709b9e790d3257c3598. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.211004 30131 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:08.211947 30132 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.212055 30132 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:07:08.212498 30132 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.212615 30132 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P 05b5f8728b794709b9e790d3257c3598: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:07:08.212821 30131 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.212939 30131 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P a5625c6033eb47aa91f1825dc77c7d64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:08.214211 30130 tablet_replica.cc:439] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05b5f8728b794709b9e790d3257c3598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } } }
I20260928 21:07:08.214329 30130 tablet_replica.cc:442] T ba3fa0c19ef145a3b9264b2b2c7e72e2 P f8cda18f67404197940f9d7276f234ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:07:08.230471 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.227041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:43413 (local address 127.21.140.65:36193)
0928 21:07:08.227176 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.227180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.227402 (+ 222us) server_negotiation.cc:407] Connection header received
0928 21:07:08.227551 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.227558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.227611 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.227701 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:08.227936 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.228483 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.229321 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.229548 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.229655 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.229914 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.230036 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.230121 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.230207 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.230291 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":55}
I20260928 21:07:08.230471 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.226980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36193 (local address 127.21.140.65:43413)
0928 21:07:08.227314 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.227329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.227458 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.227708 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.227710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.227722 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:07:08.227831 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.227837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.228601 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.228605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.229207 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.229214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.229656 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.229659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.229735 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.229782 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.230125 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.230133 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.230199 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.230293 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260928 21:07:08.236934 29909 raft_consensus.cc:1261] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Refusing update from remote peer 05b5f8728b794709b9e790d3257c3598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.236964 30049 raft_consensus.cc:1261] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Refusing update from remote peer 05b5f8728b794709b9e790d3257c3598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.237272 30135 consensus_queue.cc:1035] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.237409 30132 consensus_queue.cc:1035] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.277063 29607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:08.277266 29607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:08.279310 29607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Sending DeleteTablet for 3 replicas of tablet 2ca5737e4006499cba87710a26cc71f3
I20260928 21:07:08.279780 29745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca5737e4006499cba87710a26cc71f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45950
I20260928 21:07:08.279876 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca5737e4006499cba87710a26cc71f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45754
I20260928 21:07:08.279995 30029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca5737e4006499cba87710a26cc71f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:43642
I20260928 21:07:08.280270 30186 tablet_replica.cc:330] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: stopping tablet replica
I20260928 21:07:08.280409 30186 raft_consensus.cc:2229] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:08.280467 30185 tablet_replica.cc:330] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: stopping tablet replica
I20260928 21:07:08.280535 30186 raft_consensus.cc:2258] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.280604 30185 raft_consensus.cc:2229] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:07:08.280788 30185 raft_consensus.cc:2258] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.281294 30186 ts_tablet_manager.cc:1905] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.281375 30185 ts_tablet_manager.cc:1905] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.284192 30186 ts_tablet_manager.cc:1918] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:07:08.284245 30185 ts_tablet_manager.cc:1918] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:07:08.284281 30186 log.cc:1196] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/2ca5737e4006499cba87710a26cc71f3
I20260928 21:07:08.284317 30185 log.cc:1196] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/2ca5737e4006499cba87710a26cc71f3
I20260928 21:07:08.284554 30186 ts_tablet_manager.cc:1939] T 2ca5737e4006499cba87710a26cc71f3 P f8cda18f67404197940f9d7276f234ac: Deleting consensus metadata
I20260928 21:07:08.284615 30185 ts_tablet_manager.cc:1939] T 2ca5737e4006499cba87710a26cc71f3 P 05b5f8728b794709b9e790d3257c3598: Deleting consensus metadata
I20260928 21:07:08.285303 29592 catalog_manager.cc:4841] TS 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193): tablet 2ca5737e4006499cba87710a26cc71f3 (table test-table [id=cd834cb7a20f494d9dd1b5f7cdbfc8bc]) successfully deleted
I20260928 21:07:08.285449 30187 tablet_replica.cc:330] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: stopping tablet replica
I20260928 21:07:08.285303 29594 catalog_manager.cc:4841] TS f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589): tablet 2ca5737e4006499cba87710a26cc71f3 (table test-table [id=cd834cb7a20f494d9dd1b5f7cdbfc8bc]) successfully deleted
I20260928 21:07:08.285588 30187 raft_consensus.cc:2229] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:08.285723 30187 raft_consensus.cc:2258] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.286334 30187 ts_tablet_manager.cc:1905] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.288854 30187 ts_tablet_manager.cc:1918] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:07:08.288949 30187 log.cc:1196] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/2ca5737e4006499cba87710a26cc71f3
I20260928 21:07:08.289243 30187 ts_tablet_manager.cc:1939] T 2ca5737e4006499cba87710a26cc71f3 P a5625c6033eb47aa91f1825dc77c7d64: Deleting consensus metadata
I20260928 21:07:08.290045 29593 catalog_manager.cc:4841] TS a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183): tablet 2ca5737e4006499cba87710a26cc71f3 (table test-table [id=cd834cb7a20f494d9dd1b5f7cdbfc8bc]) successfully deleted
Sep 28 21:07:08 dist-test-slave-d5wc krb5kdc[29563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:07:08 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:08 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:07:08.309321 30160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:08.302310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.21.140.126:44773)
0928 21:07:08.302474 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.302479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.302620 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:07:08.302739 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.302742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.302802 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.302889 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:08.303561 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.304371 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.305198 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.305540 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.305693 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.309195 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41958: BlockingRecv error: recv got EOF from 127.0.0.1:41958 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 21:07:08 dist-test-slave-d5wc krb5kdc[29563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM
I20260928 21:07:08.320503 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.311214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.21.140.126:44773)
0928 21:07:08.311324 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.311328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.311539 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:07:08.311714 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.311717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.311770 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.311829 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:08.312486 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.313101 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.314251 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.314603 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.314721 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.317204 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:08.317208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:08.317211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:08.317244 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:08.318795 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:08.319227 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:08.319232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:08.319233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:08.319318 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:08.319610 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:08.319615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:08.319617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:08.319797 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:08.319937 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.320180 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.320307 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:07:08.323136 29607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:07:08.323611 29607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:08.328663 29745 tablet_service.cc:1467] Processing CreateTablet for tablet a0734379169c4b38a21480aa0cea523f (DEFAULT_TABLE table=test-table [id=0b9e5823a6dc48c7bdf36234b3b7a252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:08.328760 30029 tablet_service.cc:1467] Processing CreateTablet for tablet a0734379169c4b38a21480aa0cea523f (DEFAULT_TABLE table=test-table [id=0b9e5823a6dc48c7bdf36234b3b7a252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:08.328991 29745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0734379169c4b38a21480aa0cea523f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:08.328998 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0734379169c4b38a21480aa0cea523f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:08.330425 29888 tablet_service.cc:1467] Processing CreateTablet for tablet a0734379169c4b38a21480aa0cea523f (DEFAULT_TABLE table=test-table [id=0b9e5823a6dc48c7bdf36234b3b7a252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:08.330689 29888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0734379169c4b38a21480aa0cea523f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:08.331559 30124 tablet_bootstrap.cc:492] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Bootstrap starting.
I20260928 21:07:08.332072 30128 tablet_bootstrap.cc:492] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Bootstrap starting.
I20260928 21:07:08.332434 30124 tablet_bootstrap.cc:654] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:08.333004 30128 tablet_bootstrap.cc:654] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:08.333096 30125 tablet_bootstrap.cc:492] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Bootstrap starting.
I20260928 21:07:08.333889 30125 tablet_bootstrap.cc:654] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:08.334069 30128 tablet_bootstrap.cc:492] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: No bootstrap required, opened a new log
I20260928 21:07:08.334167 30128 ts_tablet_manager.cc:1397] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:08.334605 30128 raft_consensus.cc:348] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.334735 30128 raft_consensus.cc:374] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:08.334769 30128 raft_consensus.cc:729] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Initialized, Role: FOLLOWER
I20260928 21:07:08.334990 30128 consensus_queue.cc:260] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.335335 30125 tablet_bootstrap.cc:492] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: No bootstrap required, opened a new log
I20260928 21:07:08.335355 30128 ts_tablet_manager.cc:1428] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:07:08.335403 30125 ts_tablet_manager.cc:1397] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:07:08.335793 30125 raft_consensus.cc:348] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.335932 30125 raft_consensus.cc:374] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:08.335975 30125 raft_consensus.cc:729] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cda18f67404197940f9d7276f234ac, State: Initialized, Role: FOLLOWER
I20260928 21:07:08.336119 30125 consensus_queue.cc:260] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.336380 30125 ts_tablet_manager.cc:1428] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:07:08.337070 30124 tablet_bootstrap.cc:492] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: No bootstrap required, opened a new log
I20260928 21:07:08.337186 30124 ts_tablet_manager.cc:1397] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 21:07:08.337596 30124 raft_consensus.cc:348] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.337715 30124 raft_consensus.cc:374] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:08.337766 30124 raft_consensus.cc:729] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5625c6033eb47aa91f1825dc77c7d64, State: Initialized, Role: FOLLOWER
I20260928 21:07:08.337894 30124 consensus_queue.cc:260] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.338207 30124 ts_tablet_manager.cc:1428] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:07:08.717777 30130 raft_consensus.cc:482] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:08.717912 30130 raft_consensus.cc:504] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.719094 30130 leader_election.cc:288] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:08.724725 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.719444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36193 (local address 127.21.140.66:38729)
0928 21:07:08.719739 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.719757 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.719882 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.720219 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.720223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.720267 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:08.720513 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.720520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.721651 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.721657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.723267 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.723281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.724318 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.724321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.724415 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.724437 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.724503 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.724567 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":170,"spinlock_wait_cycles":28928,"thread_start_us":78,"threads_started":1}
I20260928 21:07:08.724762 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.719552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:38729 (local address 127.21.140.65:36193)
0928 21:07:08.719767 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.719771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.719828 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:07:08.720008 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.720012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.720067 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.720156 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:08.720638 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.721488 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.723453 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.724178 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.724343 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.724413 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.724547 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.724630 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 21:07:08.725440 29766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734379169c4b38a21480aa0cea523f" candidate_uuid: "f8cda18f67404197940f9d7276f234ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b5f8728b794709b9e790d3257c3598" is_pre_election: true
I20260928 21:07:08.725505 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.719444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38183 (local address 127.21.140.66:52607)
0928 21:07:08.719833 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.719845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.719940 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.720225 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.720227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.720268 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:08.720513 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.720520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.722026 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.722029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.723275 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.723281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.725002 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.725005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.725145 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.725178 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.725275 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.725349 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":202,"spinlock_wait_cycles":13056,"thread_start_us":117,"threads_started":1}
I20260928 21:07:08.725505 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.719547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:52607 (local address 127.21.140.67:38183)
0928 21:07:08.719790 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.719796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.719924 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:07:08.720020 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.720023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.720067 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.720156 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:08.720638 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.721476 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.723453 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.724749 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.724873 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.724931 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.725297 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.725375 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260928 21:07:08.725708 29766 raft_consensus.cc:2454] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8cda18f67404197940f9d7276f234ac in term 0.
I20260928 21:07:08.726009 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734379169c4b38a21480aa0cea523f" candidate_uuid: "f8cda18f67404197940f9d7276f234ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64" is_pre_election: true
I20260928 21:07:08.726152 30049 raft_consensus.cc:2454] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8cda18f67404197940f9d7276f234ac in term 0.
I20260928 21:07:08.726195 29839 leader_election.cc:302] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b5f8728b794709b9e790d3257c3598, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:08.726481 30130 raft_consensus.cc:2781] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:08.726544 30130 raft_consensus.cc:482] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:08.726593 30130 raft_consensus.cc:3037] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.727710 30130 raft_consensus.cc:504] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.728113 30130 leader_election.cc:288] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [CANDIDATE]: Term 1 election: Requested vote from peers 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193), a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183)
I20260928 21:07:08.728412 29766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734379169c4b38a21480aa0cea523f" candidate_uuid: "f8cda18f67404197940f9d7276f234ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b5f8728b794709b9e790d3257c3598"
I20260928 21:07:08.728506 30049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734379169c4b38a21480aa0cea523f" candidate_uuid: "f8cda18f67404197940f9d7276f234ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5625c6033eb47aa91f1825dc77c7d64"
I20260928 21:07:08.728574 29766 raft_consensus.cc:3037] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.728607 30049 raft_consensus.cc:3037] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.729732 30049 raft_consensus.cc:2454] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8cda18f67404197940f9d7276f234ac in term 1.
I20260928 21:07:08.729733 29766 raft_consensus.cc:2454] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8cda18f67404197940f9d7276f234ac in term 1.
I20260928 21:07:08.730049 29840 leader_election.cc:302] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5625c6033eb47aa91f1825dc77c7d64, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:08.730271 30130 raft_consensus.cc:2781] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:08.730484 30130 raft_consensus.cc:686] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 1 LEADER]: Becoming Leader. State: Replica: f8cda18f67404197940f9d7276f234ac, State: Running, Role: LEADER
I20260928 21:07:08.730739 30130 consensus_queue.cc:237] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.733286 29607 catalog_manager.cc:5495] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac reported cstate change: term changed from 0 to 1, leader changed from <none> to f8cda18f67404197940f9d7276f234ac (127.21.140.66). New cstate: current_term: 1 leader_uuid: "f8cda18f67404197940f9d7276f234ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } health_report { overall_health: UNKNOWN } } }
I20260928 21:07:08.795626 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.791897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.21.140.66:40589)
0928 21:07:08.792174 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.792178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.792190 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:07:08.792263 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.792265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.792318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.792426 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:08.792657 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.793169 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.794101 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.794347 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.794464 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.794764 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.795226 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.795320 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.795449 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.795500 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
I20260928 21:07:08.798480 30049 raft_consensus.cc:1261] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Refusing update from remote peer f8cda18f67404197940f9d7276f234ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.798743 29766 raft_consensus.cc:1261] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Refusing update from remote peer f8cda18f67404197940f9d7276f234ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.799024 30130 consensus_queue.cc:1035] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.799336 30206 consensus_queue.cc:1035] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.810515 29607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:08.810653 29607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:08.812687 29607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Sending DeleteTablet for 3 replicas of tablet a0734379169c4b38a21480aa0cea523f
I20260928 21:07:08.813205 29745 tablet_service.cc:1514] Processing DeleteTablet for tablet a0734379169c4b38a21480aa0cea523f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45950
I20260928 21:07:08.813271 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet a0734379169c4b38a21480aa0cea523f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45754
I20260928 21:07:08.813442 30029 tablet_service.cc:1514] Processing DeleteTablet for tablet a0734379169c4b38a21480aa0cea523f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:43642
I20260928 21:07:08.813766 30220 tablet_replica.cc:330] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: stopping tablet replica
I20260928 21:07:08.813951 30221 tablet_replica.cc:330] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: stopping tablet replica
I20260928 21:07:08.814101 30221 raft_consensus.cc:2229] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:08.814141 30219 tablet_replica.cc:330] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: stopping tablet replica
I20260928 21:07:08.814234 30221 raft_consensus.cc:2258] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.814256 30219 raft_consensus.cc:2229] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:08.814428 30220 raft_consensus.cc:2229] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:07:08.814383 29607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:07:08.814536 30219 raft_consensus.cc:2258] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:07:08.814818 29607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:08.814930 30220 raft_consensus.cc:2258] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.815557 30220 ts_tablet_manager.cc:1905] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.815557 30221 ts_tablet_manager.cc:1905] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.818179 30221 ts_tablet_manager.cc:1918] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:08.818179 30220 ts_tablet_manager.cc:1918] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:08.818255 30221 log.cc:1196] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/a0734379169c4b38a21480aa0cea523f
I20260928 21:07:08.818255 30220 log.cc:1196] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/a0734379169c4b38a21480aa0cea523f
I20260928 21:07:08.818533 30221 ts_tablet_manager.cc:1939] T a0734379169c4b38a21480aa0cea523f P a5625c6033eb47aa91f1825dc77c7d64: Deleting consensus metadata
I20260928 21:07:08.818531 30220 ts_tablet_manager.cc:1939] T a0734379169c4b38a21480aa0cea523f P f8cda18f67404197940f9d7276f234ac: Deleting consensus metadata
I20260928 21:07:08.819664 29594 catalog_manager.cc:4841] TS f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589): tablet a0734379169c4b38a21480aa0cea523f (table test-table [id=0b9e5823a6dc48c7bdf36234b3b7a252]) successfully deleted
I20260928 21:07:08.819728 29593 catalog_manager.cc:4841] TS a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183): tablet a0734379169c4b38a21480aa0cea523f (table test-table [id=0b9e5823a6dc48c7bdf36234b3b7a252]) successfully deleted
I20260928 21:07:08.820840 29745 tablet_service.cc:1467] Processing CreateTablet for tablet 656feac9152e492ba2ce74ed2b2d7607 (DEFAULT_TABLE table=test-table [id=7be9a78a83d44bac8162444509e81fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:08.821022 30029 tablet_service.cc:1467] Processing CreateTablet for tablet 656feac9152e492ba2ce74ed2b2d7607 (DEFAULT_TABLE table=test-table [id=7be9a78a83d44bac8162444509e81fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:08.821188 29745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656feac9152e492ba2ce74ed2b2d7607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:08.821254 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656feac9152e492ba2ce74ed2b2d7607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:08.822635 30219 ts_tablet_manager.cc:1905] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.823498 30124 tablet_bootstrap.cc:492] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Bootstrap starting.
I20260928 21:07:08.823685 30128 tablet_bootstrap.cc:492] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Bootstrap starting.
I20260928 21:07:08.824371 30124 tablet_bootstrap.cc:654] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:08.824456 30128 tablet_bootstrap.cc:654] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:08.825338 29888 tablet_service.cc:1467] Processing CreateTablet for tablet 656feac9152e492ba2ce74ed2b2d7607 (DEFAULT_TABLE table=test-table [id=7be9a78a83d44bac8162444509e81fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:08.825587 29888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656feac9152e492ba2ce74ed2b2d7607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:08.826040 30219 ts_tablet_manager.cc:1918] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:08.826118 30219 log.cc:1196] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/a0734379169c4b38a21480aa0cea523f
I20260928 21:07:08.826445 30219 ts_tablet_manager.cc:1939] T a0734379169c4b38a21480aa0cea523f P 05b5f8728b794709b9e790d3257c3598: Deleting consensus metadata
I20260928 21:07:08.827111 29592 catalog_manager.cc:4841] TS 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193): tablet a0734379169c4b38a21480aa0cea523f (table test-table [id=0b9e5823a6dc48c7bdf36234b3b7a252]) successfully deleted
I20260928 21:07:08.827811 30125 tablet_bootstrap.cc:492] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Bootstrap starting.
I20260928 21:07:08.828750 30125 tablet_bootstrap.cc:654] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:08.828933 30128 tablet_bootstrap.cc:492] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: No bootstrap required, opened a new log
I20260928 21:07:08.829056 30128 ts_tablet_manager.cc:1397] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 21:07:08.829447 30128 raft_consensus.cc:348] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.829581 30128 raft_consensus.cc:374] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:08.829628 30128 raft_consensus.cc:729] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5f8728b794709b9e790d3257c3598, State: Initialized, Role: FOLLOWER
I20260928 21:07:08.829754 30128 consensus_queue.cc:260] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.830034 30125 tablet_bootstrap.cc:492] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: No bootstrap required, opened a new log
I20260928 21:07:08.830083 30128 ts_tablet_manager.cc:1428] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:08.830116 30125 ts_tablet_manager.cc:1397] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:08.830575 30125 raft_consensus.cc:348] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.830680 30125 raft_consensus.cc:374] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:08.830698 30124 tablet_bootstrap.cc:492] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: No bootstrap required, opened a new log
I20260928 21:07:08.830716 30125 raft_consensus.cc:729] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cda18f67404197940f9d7276f234ac, State: Initialized, Role: FOLLOWER
I20260928 21:07:08.830776 30124 ts_tablet_manager.cc:1397] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 21:07:08.830832 30125 consensus_queue.cc:260] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.831079 30125 ts_tablet_manager.cc:1428] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:07:08.831249 30124 raft_consensus.cc:348] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.831357 30124 raft_consensus.cc:374] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:08.831394 30124 raft_consensus.cc:729] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5625c6033eb47aa91f1825dc77c7d64, State: Initialized, Role: FOLLOWER
I20260928 21:07:08.831519 30124 consensus_queue.cc:260] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.831779 30124 ts_tablet_manager.cc:1428] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:07:08.879364 30131 raft_consensus.cc:482] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:08.879518 30131 raft_consensus.cc:504] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.880740 30131 leader_election.cc:288] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193), f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589)
I20260928 21:07:08.885944 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.881384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:38967 (local address 127.21.140.65:36193)
0928 21:07:08.881516 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.881521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.881558 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:07:08.881723 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.881726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.881772 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.881819 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:08.882344 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.883171 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.884579 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.885180 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.885272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.885326 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.885702 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.885791 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:07:08.886112 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.881657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:47757 (local address 127.21.140.66:40589)
0928 21:07:08.881769 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.881773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.882087 (+ 314us) server_negotiation.cc:407] Connection header received
0928 21:07:08.882377 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.882385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.882431 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.882531 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:08.882802 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.883343 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.884936 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.885498 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.885617 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.885686 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.885880 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.885994 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:07:08.886489 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.881068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:40589 (local address 127.21.140.67:47757)
0928 21:07:08.882007 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.882021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.882155 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.882542 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.882545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.882562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:08.882680 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.882685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.883471 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.883474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.884769 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.884780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.885631 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.885635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.885738 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.885763 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.885844 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.885914 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":30,"thread_start_us":94,"threads_started":1}
I20260928 21:07:08.886499 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.881017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36193 (local address 127.21.140.67:38967)
0928 21:07:08.881476 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.881491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.881584 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.881956 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.881958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.882008 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:08.882149 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.882158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.883322 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.883326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.884434 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.884445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.885436 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.885439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.885546 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.885581 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.886119 (+ 538us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.886202 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260928 21:07:08.886991 29908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656feac9152e492ba2ce74ed2b2d7607" candidate_uuid: "a5625c6033eb47aa91f1825dc77c7d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac" is_pre_election: true
I20260928 21:07:08.887063 29766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656feac9152e492ba2ce74ed2b2d7607" candidate_uuid: "a5625c6033eb47aa91f1825dc77c7d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b5f8728b794709b9e790d3257c3598" is_pre_election: true
I20260928 21:07:08.887156 29908 raft_consensus.cc:2454] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5625c6033eb47aa91f1825dc77c7d64 in term 0.
I20260928 21:07:08.887233 29766 raft_consensus.cc:2454] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5625c6033eb47aa91f1825dc77c7d64 in term 0.
I20260928 21:07:08.887578 29982 leader_election.cc:302] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5625c6033eb47aa91f1825dc77c7d64, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:08.887876 30131 raft_consensus.cc:2781] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:08.887951 30131 raft_consensus.cc:482] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:08.887990 30131 raft_consensus.cc:3037] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.889030 30131 raft_consensus.cc:504] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.889420 30131 leader_election.cc:288] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [CANDIDATE]: Term 1 election: Requested vote from peers 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193), f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589)
I20260928 21:07:08.889770 29766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656feac9152e492ba2ce74ed2b2d7607" candidate_uuid: "a5625c6033eb47aa91f1825dc77c7d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b5f8728b794709b9e790d3257c3598"
I20260928 21:07:08.889788 29908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656feac9152e492ba2ce74ed2b2d7607" candidate_uuid: "a5625c6033eb47aa91f1825dc77c7d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cda18f67404197940f9d7276f234ac"
I20260928 21:07:08.889896 29766 raft_consensus.cc:3037] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.889927 29908 raft_consensus.cc:3037] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:08.890879 29908 raft_consensus.cc:2454] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5625c6033eb47aa91f1825dc77c7d64 in term 1.
I20260928 21:07:08.891085 29766 raft_consensus.cc:2454] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5625c6033eb47aa91f1825dc77c7d64 in term 1.
I20260928 21:07:08.891244 29982 leader_election.cc:302] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5625c6033eb47aa91f1825dc77c7d64, f8cda18f67404197940f9d7276f234ac; no voters:
I20260928 21:07:08.891412 30131 raft_consensus.cc:2781] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:08.891638 30131 raft_consensus.cc:686] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 LEADER]: Becoming Leader. State: Replica: a5625c6033eb47aa91f1825dc77c7d64, State: Running, Role: LEADER
I20260928 21:07:08.891940 30131 consensus_queue.cc:237] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } }
I20260928 21:07:08.894465 29607 catalog_manager.cc:5495] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "a5625c6033eb47aa91f1825dc77c7d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5625c6033eb47aa91f1825dc77c7d64" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 38183 } health_report { overall_health: HEALTHY } } }
I20260928 21:07:08.914242 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.909929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.21.140.67:38183)
0928 21:07:08.910289 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.910292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.910302 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:07:08.910383 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.910385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.910478 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.910557 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:08.910830 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.911420 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.912557 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.912772 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.912858 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.913218 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.913690 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.913783 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.913928 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.914053 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260928 21:07:08.925357 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.921542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:58663 (local address 127.21.140.65:36193)
0928 21:07:08.921673 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.921676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.921811 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:07:08.922003 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.922008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.922059 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.922128 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:08.922434 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.922906 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.924208 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.924437 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.924520 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.924916 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.924993 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.925061 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.925193 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.925248 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:07:08.925436 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.921377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36193 (local address 127.21.140.67:58663)
0928 21:07:08.921719 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.921734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.921843 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.922170 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.922174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.922182 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:07:08.922309 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.922315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.923055 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.923058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.924060 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.924071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.924554 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.924558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.924658 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.924710 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.925093 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.925101 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.925169 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.925249 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260928 21:07:08.931599 29766 raft_consensus.cc:1261] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Refusing update from remote peer a5625c6033eb47aa91f1825dc77c7d64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.931768 29908 raft_consensus.cc:1261] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Refusing update from remote peer a5625c6033eb47aa91f1825dc77c7d64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:08.932241 30131 consensus_queue.cc:1035] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b5f8728b794709b9e790d3257c3598" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.932435 30227 consensus_queue.cc:1035] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cda18f67404197940f9d7276f234ac" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:08.953385 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.948539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:38183 (local address 127.21.140.65:44267)
0928 21:07:08.949165 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:08.949179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:08.949319 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:08.949687 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:08.949691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:08.949702 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:07:08.949831 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.949839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.950819 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.950822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.951661 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:08.951671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.952532 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.952537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:08.952630 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.952662 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.953077 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.953087 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:07:08.953176 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:07:08.953251 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":107,"threads_started":1}
I20260928 21:07:08.953406 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:08.948858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:44267 (local address 127.21.140.67:38183)
0928 21:07:08.948977 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:08.948984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:08.949245 (+ 261us) server_negotiation.cc:407] Connection header received
0928 21:07:08.949440 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:08.949443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:08.949501 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:08.949549 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:08.949989 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.950691 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.951955 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:08.952297 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:08.952411 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:08.952777 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:08.952867 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:08.952956 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:08.953213 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 21:07:08.953277 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:07:08.981248 29606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:08.981439 29606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:08.983592 29606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb6e153538504971bf53f45ac933a939: Sending DeleteTablet for 3 replicas of tablet 656feac9152e492ba2ce74ed2b2d7607
I20260928 21:07:08.984197 29745 tablet_service.cc:1514] Processing DeleteTablet for tablet 656feac9152e492ba2ce74ed2b2d7607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45950
I20260928 21:07:08.984400 30219 tablet_replica.cc:330] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: stopping tablet replica
I20260928 21:07:08.984524 30219 raft_consensus.cc:2229] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:08.984611 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet 656feac9152e492ba2ce74ed2b2d7607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:45754
I20260928 21:07:08.984647 30219 raft_consensus.cc:2258] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.984787 30029 tablet_service.cc:1514] Processing DeleteTablet for tablet 656feac9152e492ba2ce74ed2b2d7607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:08 UTC) from {username='oryx', principal='oryx/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:43642
I20260928 21:07:08.984956 30221 tablet_replica.cc:330] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: stopping tablet replica
I20260928 21:07:08.984966 30220 tablet_replica.cc:330] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: stopping tablet replica
I20260928 21:07:08.985069 30221 raft_consensus.cc:2229] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:07:08.985072 30220 raft_consensus.cc:2229] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:08.985173 30220 raft_consensus.cc:2258] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.985379 30221 raft_consensus.cc:2258] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:08.986086 30220 ts_tablet_manager.cc:1905] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.986359 30219 ts_tablet_manager.cc:1905] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.988654 30220 ts_tablet_manager.cc:1918] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:07:08.988720 30220 log.cc:1196] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/656feac9152e492ba2ce74ed2b2d7607
I20260928 21:07:08.988967 30220 ts_tablet_manager.cc:1939] T 656feac9152e492ba2ce74ed2b2d7607 P f8cda18f67404197940f9d7276f234ac: Deleting consensus metadata
I20260928 21:07:08.989087 30219 ts_tablet_manager.cc:1918] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:07:08.989164 30219 log.cc:1196] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/656feac9152e492ba2ce74ed2b2d7607
I20260928 21:07:08.989423 30219 ts_tablet_manager.cc:1939] T 656feac9152e492ba2ce74ed2b2d7607 P 05b5f8728b794709b9e790d3257c3598: Deleting consensus metadata
I20260928 21:07:08.989647 29594 catalog_manager.cc:4841] TS f8cda18f67404197940f9d7276f234ac (127.21.140.66:40589): tablet 656feac9152e492ba2ce74ed2b2d7607 (table test-table [id=7be9a78a83d44bac8162444509e81fd4]) successfully deleted
I20260928 21:07:08.990193 29592 catalog_manager.cc:4841] TS 05b5f8728b794709b9e790d3257c3598 (127.21.140.65:36193): tablet 656feac9152e492ba2ce74ed2b2d7607 (table test-table [id=7be9a78a83d44bac8162444509e81fd4]) successfully deleted
I20260928 21:07:08.990556 30221 ts_tablet_manager.cc:1905] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:08.992921 30221 ts_tablet_manager.cc:1918] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:07:08.992990 30221 log.cc:1196] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/656feac9152e492ba2ce74ed2b2d7607
I20260928 21:07:08.993222 30221 ts_tablet_manager.cc:1939] T 656feac9152e492ba2ce74ed2b2d7607 P a5625c6033eb47aa91f1825dc77c7d64: Deleting consensus metadata
I20260928 21:07:08.993999 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 29681
I20260928 21:07:08.994896 29593 catalog_manager.cc:4841] TS a5625c6033eb47aa91f1825dc77c7d64 (127.21.140.67:38183): tablet 656feac9152e492ba2ce74ed2b2d7607 (table test-table [id=7be9a78a83d44bac8162444509e81fd4]) successfully deleted
I20260928 21:07:09.004269 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 29823
I20260928 21:07:09.011947 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 29964
I20260928 21:07:09.018111 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 29576
2026-09-28T21:07:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3824 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:07:09 dist-test-slave-d5wc krb5kdc[30243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:07:09 dist-test-slave-d5wc krb5kdc[30243](info): set up 2 sockets
Sep 28 21:07:09 dist-test-slave-d5wc krb5kdc[30243](info): commencing operation
krb5kdc: starting...
W20260928 21:07:10.064285 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:07:10 dist-test-slave-d5wc krb5kdc[30243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:07:10.189843 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44403
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:10.300016 30256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:10.300272 30256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:10.300338 30256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:10.304370 30256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:10.304421 30256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:10.304440 30256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:10.304456 30256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:10.304502 30256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:10.309803 30256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:10.310835 30256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:10.311820 30256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:10.320514 30261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:10.320523 30256 server_base.cc:1034] running on GCE node
W20260928 21:07:10.320519 30264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:10.320607 30262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:10.321306 30256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:10.322444 30256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:10.323654 30256 hybrid_clock.cc:648] HybridClock initialized: now 1790629630323636 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T21:07:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1319 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:07:10 dist-test-slave-d5wc krb5kdc[30269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:07:10 dist-test-slave-d5wc krb5kdc[30269](info): set up 2 sockets
Sep 28 21:07:10 dist-test-slave-d5wc krb5kdc[30269](info): commencing operation
krb5kdc: starting...
W20260928 21:07:11.385464 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:07:11 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:07:11.506795 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37743
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:11.615477 30282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:11.615780 30282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:11.615852 30282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:11.619838 30282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:11.619941 30282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:11.619982 30282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:11.620041 30282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:11.620080 30282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:11.625329 30282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:11.626359 30282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:11.627333 30282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:11.635468 30288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:11.635493 30287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:11.635612 30282 server_base.cc:1034] running on GCE node
W20260928 21:07:11.635782 30290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:11.636286 30282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:11.637454 30282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:11.638648 30282 hybrid_clock.cc:648] HybridClock initialized: now 1790629631638636 us; error 48 us; skew 500 ppm
Sep 28 21:07:11 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629631, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:11.641404 30282 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:07:11.642328 30282 webserver.cc:460] Webserver started at http://127.21.140.126:40947/ using document root <none> and password file <none>
I20260928 21:07:11.642894 30282 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:11.642956 30282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:11.643184 30282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:11.645222 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "dfcf53185d1c450d920bcbe0735776d3"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "83b0e64516dc26fc53439449b80f9673"
server_key_iv: "3a51ff4b1ca7c902da2dd1f22c8a1c71"
server_key_version: "encryptionkey@0"
I20260928 21:07:11.645815 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "dfcf53185d1c450d920bcbe0735776d3"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "83b0e64516dc26fc53439449b80f9673"
server_key_iv: "3a51ff4b1ca7c902da2dd1f22c8a1c71"
server_key_version: "encryptionkey@0"
I20260928 21:07:11.649495 30282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:11.651966 30297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:11.653206 30282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:07:11.653362 30282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "dfcf53185d1c450d920bcbe0735776d3"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "83b0e64516dc26fc53439449b80f9673"
server_key_iv: "3a51ff4b1ca7c902da2dd1f22c8a1c71"
server_key_version: "encryptionkey@0"
I20260928 21:07:11.653481 30282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:11.664989 30282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:11.667842 30282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:11.668046 30282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:11.675419 30282 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37743
I20260928 21:07:11.675439 30349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37743 every 8 connection(s)
I20260928 21:07:11.676373 30282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:07:11.679091 30350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:11.682792 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 30282
I20260928 21:07:11.682897 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:07:11.683192 22065 external_mini_cluster.cc:1428] Setting key a99acc6f3cf60cd67969be639225bc59
I20260928 21:07:11.685076 30350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Bootstrap starting.
I20260928 21:07:11.687348 30350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:11.688094 30350 log.cc:824] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:11.689888 30350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: No bootstrap required, opened a new log
Sep 28 21:07:11 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:07:11.693738 30350 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:11.693954 30350 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:11.694061 30350 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcf53185d1c450d920bcbe0735776d3, State: Initialized, Role: FOLLOWER
I20260928 21:07:11.694478 30350 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:11.694597 30350 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:07:11.694682 30350 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:07:11.694783 30350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:11.695930 30350 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:11.696296 30350 leader_election.cc:302] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcf53185d1c450d920bcbe0735776d3; no voters:
I20260928 21:07:11.696635 30350 leader_election.cc:288] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:07:11.696798 30355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:11.697026 30355 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 LEADER]: Becoming Leader. State: Replica: dfcf53185d1c450d920bcbe0735776d3, State: Running, Role: LEADER
I20260928 21:07:11.697376 30355 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:11.697861 30350 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:11.698343 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:11.684613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.21.140.126:37743)
0928 21:07:11.685048 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:11.685063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:11.685120 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:07:11.685796 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:11.685813 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:11.686218 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:11.686551 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:11.687374 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:11.688450 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:11.689320 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:11.689784 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:11.689958 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:11.692759 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:11.692763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:11.692776 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:11.692802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:11.694875 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:11.695351 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:11.695355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:11.695360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:11.695463 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:11.695752 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:11.695754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:11.695755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:11.696191 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:11.696341 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:11.696676 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 21:07:11.697376 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260928 21:07:11.698585 30355 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcf53185d1c450d920bcbe0735776d3. Latest consensus state: current_term: 1 leader_uuid: "dfcf53185d1c450d920bcbe0735776d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } } }
I20260928 21:07:11.698704 30355 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:11.699106 30363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:11.699352 30356 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfcf53185d1c450d920bcbe0735776d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } } }
I20260928 21:07:11.699455 30356 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:11.702296 30363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:11.708284 30363 catalog_manager.cc:1347] Generated new cluster ID: 47f574c3b5b44bb29ec97122e93e0592
I20260928 21:07:11.708356 30363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:11.715790 30363 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:07:11.716712 30363 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:11.730911 30363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Generated new TSK 0
I20260928 21:07:11.731844 30363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:07:11.769707 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37743
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:11.881107 30376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:11.881407 30376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:11.881484 30376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:11.885509 30376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:11.885612 30376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:11.885710 30376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:11.891088 30376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:11.892202 30376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:11.893249 30376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:11.903267 30384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:11.903255 30382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:11.903414 30376 server_base.cc:1034] running on GCE node
W20260928 21:07:11.903439 30381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:11.903911 30376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:11.904583 30376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:11.905771 30376 hybrid_clock.cc:648] HybridClock initialized: now 1790629631905754 us; error 30 us; skew 500 ppm
Sep 28 21:07:11 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629631, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:11.908653 30376 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:07:11.909571 30376 webserver.cc:460] Webserver started at http://127.21.140.65:43367/ using document root <none> and password file <none>
I20260928 21:07:11.910154 30376 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:11.910223 30376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:11.910388 30376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:11.912591 30376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "ed5bb1138301410fa70103417ada2b2e"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "5496666436496f9f664b8bb6bf3dde0c"
server_key_iv: "5c63fc1e144a55fbb7add97bd83e8822"
server_key_version: "encryptionkey@0"
I20260928 21:07:11.913256 30376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "ed5bb1138301410fa70103417ada2b2e"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "5496666436496f9f664b8bb6bf3dde0c"
server_key_iv: "5c63fc1e144a55fbb7add97bd83e8822"
server_key_version: "encryptionkey@0"
I20260928 21:07:11.916989 30376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:07:11.919503 30392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:11.920792 30376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:07:11.920904 30376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "ed5bb1138301410fa70103417ada2b2e"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "5496666436496f9f664b8bb6bf3dde0c"
server_key_iv: "5c63fc1e144a55fbb7add97bd83e8822"
server_key_version: "encryptionkey@0"
I20260928 21:07:11.920994 30376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:11.941395 30376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:11.944324 30376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:11.944540 30376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:11.945130 30376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:11.945945 30376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:11.946110 30376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:11.946180 30376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:11.946231 30376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:11.955598 30376 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:34305
I20260928 21:07:11.955626 30506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:34305 every 8 connection(s)
I20260928 21:07:11.956578 30376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:11.966375 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 30376
I20260928 21:07:11.966524 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:07:11.966818 22065 external_mini_cluster.cc:1428] Setting key 7ebc4c4e1c6345b54c61a19c9517f426
Sep 28 21:07:11 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629631, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:07:11.971566 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:11.958715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:36477 (local address 127.21.140.126:37743)
0928 21:07:11.958870 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:11.958875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:11.960040 (+ 1165us) server_negotiation.cc:407] Connection header received
0928 21:07:11.961527 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:11.961530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:11.961579 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:11.961660 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:11.962893 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:11.963447 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:11.964600 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:11.964827 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:11.964922 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:11.967532 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:11.967536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:11.967539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:11.967566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:11.969360 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:11.969879 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:11.969886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:11.969890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:11.970002 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:11.970422 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:11.970424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:11.970425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:11.970600 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:11.970677 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:11.971330 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 21:07:11.971449 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:07:11.972414 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:11.959308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:36477)
0928 21:07:11.959853 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:11.959915 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:11.961303 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:11.961793 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:11.961800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:11.962253 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:11.962733 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:11.962747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:11.963583 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:11.963585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:11.964454 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:11.964462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:11.964970 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:11.964973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:11.965128 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:11.965684 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:11.965700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:11.967363 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:11.969509 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:11.969514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:11.969526 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:11.969778 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:11.970105 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:11.970111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:11.970113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:11.970315 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:11.970708 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:11.970718 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:11.971051 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 21:07:11.971288 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 21:07:11.971576 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260928 21:07:11.973762 30507 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37743
I20260928 21:07:11.974077 30507 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:11.974749 30507 heartbeater.cc:511] Master 127.21.140.126:37743 requested a full tablet report, sending...
I20260928 21:07:11.976639 30314 ts_manager.cc:194] Registered new tserver with Master: ed5bb1138301410fa70103417ada2b2e (127.21.140.65:34305)
I20260928 21:07:11.977919 30314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:36477
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:07:11.996814 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37743
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:12.107663 30512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:12.107987 30512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:12.108072 30512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:12.112188 30512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:12.112308 30512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:12.112409 30512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:12.117921 30512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:12.119138 30512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:12.120198 30512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:12.130149 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.130151 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:12.130236 30512 server_base.cc:1034] running on GCE node
W20260928 21:07:12.130146 30520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:12.130824 30512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:12.131419 30512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:12.132642 30512 hybrid_clock.cc:648] HybridClock initialized: now 1790629632132623 us; error 31 us; skew 500 ppm
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629632, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:12.135526 30512 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:07:12.136422 30512 webserver.cc:460] Webserver started at http://127.21.140.66:34395/ using document root <none> and password file <none>
I20260928 21:07:12.136991 30512 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:12.137079 30512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:12.137312 30512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:12.139364 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "ab80259eeb634d55b6015e96e89f80b7"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "d137d01a34782cd2e2903b50d422cc36"
server_key_iv: "773550f101db560ea85dc663a7e17585"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.139930 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "ab80259eeb634d55b6015e96e89f80b7"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "d137d01a34782cd2e2903b50d422cc36"
server_key_iv: "773550f101db560ea85dc663a7e17585"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.143709 30512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:07:12.146179 30527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.147490 30512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:07:12.147630 30512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "ab80259eeb634d55b6015e96e89f80b7"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "d137d01a34782cd2e2903b50d422cc36"
server_key_iv: "773550f101db560ea85dc663a7e17585"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.147746 30512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:12.172873 30512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:12.176263 30512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:12.176496 30512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:12.177088 30512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:12.177891 30512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:12.177966 30512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.178120 30512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:12.178174 30512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.187479 30512 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:42345
I20260928 21:07:12.187496 30640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:42345 every 8 connection(s)
I20260928 21:07:12.188344 30512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:12.193696 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 30512
I20260928 21:07:12.193804 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:07:12.194190 22065 external_mini_cluster.cc:1428] Setting key fb1dfa301e5206f8c8ba117afe08e61c
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629632, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:07:12.203969 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.190108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:34831 (local address 127.21.140.126:37743)
0928 21:07:12.190294 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:12.190298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:12.191115 (+ 817us) server_negotiation.cc:407] Connection header received
0928 21:07:12.192407 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:12.192411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:12.192470 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:12.192609 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:12.193909 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.194732 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.196234 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.196644 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.196773 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.199820 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:12.199823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:12.199826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:12.199849 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:12.201382 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:12.202089 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:12.202092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:12.202093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:12.202170 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:12.202658 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:12.202661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:12.202663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:12.202921 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:12.203092 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:12.203650 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 21:07:12.203782 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:07:12.204782 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.190534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:34831)
0928 21:07:12.190951 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:12.191001 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:12.192187 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:12.192787 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:12.192794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:12.193168 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:12.193704 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.193718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.194886 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.194889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.196098 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.196108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.196810 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.196813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.196954 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.197677 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:12.197694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:12.199632 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:12.201514 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:12.201523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:12.201543 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:12.201941 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:12.202272 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:12.202277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:12.202280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:12.202486 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:12.203103 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:12.203112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:12.203392 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 21:07:12.203589 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:07:12.203918 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:07:12.206209 30641 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37743
I20260928 21:07:12.206545 30641 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:12.207104 30641 heartbeater.cc:511] Master 127.21.140.126:37743 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:07:12.208416 30314 ts_manager.cc:194] Registered new tserver with Master: ab80259eeb634d55b6015e96e89f80b7 (127.21.140.66:42345)
I20260928 21:07:12.209259 30314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:34831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:07:12.225076 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37743
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:12.334604 30646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:12.334888 30646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:12.334985 30646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:12.339076 30646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:12.339157 30646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:12.339243 30646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:12.345082 30646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:12.346261 30646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:12.347286 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:07:12.357206 30646 server_base.cc:1034] running on GCE node
W20260928 21:07:12.357158 30652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.357112 30651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.357112 30654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:12.357834 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:12.358528 30646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:12.359758 30646 hybrid_clock.cc:648] HybridClock initialized: now 1790629632359724 us; error 49 us; skew 500 ppm
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629632, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:12.362675 30646 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:07:12.363519 30646 webserver.cc:460] Webserver started at http://127.21.140.67:42511/ using document root <none> and password file <none>
I20260928 21:07:12.364040 30646 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:12.364097 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:12.364256 30646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:12.366325 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "b4a9d1c80bd6432d916c5ff044b01cbb"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "df6bfdd6e5bda66dea5b7bd1bc190869"
server_key_iv: "136865b541c6da23dfe13d2fadfb2c89"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.366820 30646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "b4a9d1c80bd6432d916c5ff044b01cbb"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "df6bfdd6e5bda66dea5b7bd1bc190869"
server_key_iv: "136865b541c6da23dfe13d2fadfb2c89"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.370481 30646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:07:12.372965 30661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.374325 30646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:12.374455 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "b4a9d1c80bd6432d916c5ff044b01cbb"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "df6bfdd6e5bda66dea5b7bd1bc190869"
server_key_iv: "136865b541c6da23dfe13d2fadfb2c89"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.374571 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:12.396414 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:12.400068 30646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:12.400280 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:12.400918 30646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:12.401780 30646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:12.401827 30646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.401868 30646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:12.401922 30646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.410800 30646 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:41903
I20260928 21:07:12.410931 30774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:41903 every 8 connection(s)
I20260928 21:07:12.411800 30646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629632, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:07:12.422181 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 30646
I20260928 21:07:12.422320 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:07:12.422614 22065 external_mini_cluster.cc:1428] Setting key f541d7fccf978c47c07151fb96332243
I20260928 21:07:12.425645 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.413695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:47995 (local address 127.21.140.126:37743)
0928 21:07:12.413836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:12.413840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:12.414774 (+ 934us) server_negotiation.cc:407] Connection header received
0928 21:07:12.416017 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:12.416020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:12.416067 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:12.416147 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:12.417321 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.417864 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.419042 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.419265 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.419379 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.421840 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:12.421844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:12.421846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:12.421869 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:12.423586 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:12.424073 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:12.424077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:12.424078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:12.424203 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:12.424514 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:12.424517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:12.424519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:12.424747 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:12.424870 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:12.425348 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 21:07:12.425480 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:07:12.426486 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.414162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:47995)
0928 21:07:12.414614 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:12.414657 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:12.415802 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:12.416308 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:12.416315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:12.416698 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:12.417172 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.417184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.418034 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.418037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.418897 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.418905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.419416 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.419420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.419605 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.420207 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:12.420223 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:12.421661 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:12.423684 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:12.423690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:12.423704 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:12.423957 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:12.424297 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:12.424299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:12.424300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:12.424424 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:12.424866 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:12.424872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:12.425119 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 21:07:12.425306 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:07:12.425615 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260928 21:07:12.427559 30775 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37743
I20260928 21:07:12.427763 30775 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:12.428226 30775 heartbeater.cc:511] Master 127.21.140.126:37743 requested a full tablet report, sending...
I20260928 21:07:12.429154 30314 ts_manager.cc:194] Registered new tserver with Master: b4a9d1c80bd6432d916c5ff044b01cbb (127.21.140.67:41903)
I20260928 21:07:12.429675 30314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:47995
I20260928 21:07:12.436827 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab.
Entry for principal oryx/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab.
WARNING: no policy specified for oryx/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab.
Entry for principal oryx/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab.
WARNING: no policy specified for oryx/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab.
Entry for principal oryx/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab.
I20260928 21:07:12.513510 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 30376
I20260928 21:07:12.518790 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 30512
I20260928 21:07:12.524333 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 30646
I20260928 21:07:12.529594 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 30282
I20260928 21:07:12.536437 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37743
--webserver_interface=127.21.140.126
--webserver_port=40947
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:12.647791 30784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:12.648073 30784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:12.648116 30784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:12.652233 30784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:12.652292 30784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:12.652312 30784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:12.652329 30784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:12.652344 30784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:12.657655 30784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:12.658726 30784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:12.659731 30784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:12.668411 30789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.668443 30792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.668416 30790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:12.668807 30784 server_base.cc:1034] running on GCE node
I20260928 21:07:12.669404 30784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:12.670554 30784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:12.671746 30784 hybrid_clock.cc:648] HybridClock initialized: now 1790629632671740 us; error 31 us; skew 500 ppm
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629632, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:12.674587 30784 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:07:12.675467 30784 webserver.cc:460] Webserver started at http://127.21.140.126:40947/ using document root <none> and password file <none>
I20260928 21:07:12.676020 30784 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:12.676129 30784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:12.679996 30784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:07:12.682168 30799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.683427 30784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:12.683570 30784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "dfcf53185d1c450d920bcbe0735776d3"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "83b0e64516dc26fc53439449b80f9673"
server_key_iv: "3a51ff4b1ca7c902da2dd1f22c8a1c71"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.684113 30784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:12.694425 30784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:12.697394 30784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:12.697599 30784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:12.704847 30784 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37743
I20260928 21:07:12.704865 30851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37743 every 8 connection(s)
I20260928 21:07:12.705958 30784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:07:12.710224 30852 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:07:12.711817 30852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Bootstrap starting.
I20260928 21:07:12.713080 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 30784
I20260928 21:07:12.713590 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:34305
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:37743
--webserver_port=43367
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:07:12.729393 30852 log.cc:824] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:12.736220 30852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:07:12.736653 30852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Bootstrap complete.
I20260928 21:07:12.741014 30852 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:12.741441 30852 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcf53185d1c450d920bcbe0735776d3, State: Initialized, Role: FOLLOWER
I20260928 21:07:12.742060 30852 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:12.742185 30852 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:07:12.742273 30852 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:07:12.742374 30852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:07:12.744390 30852 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:12.744767 30852 leader_election.cc:302] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcf53185d1c450d920bcbe0735776d3; no voters:
I20260928 21:07:12.745283 30852 leader_election.cc:288] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:07:12.745397 30856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:07:12.745611 30856 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [term 2 LEADER]: Becoming Leader. State: Replica: dfcf53185d1c450d920bcbe0735776d3, State: Running, Role: LEADER
I20260928 21:07:12.745939 30856 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } }
I20260928 21:07:12.746657 30852 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:12.746987 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcf53185d1c450d920bcbe0735776d3. Latest consensus state: current_term: 2 leader_uuid: "dfcf53185d1c450d920bcbe0735776d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } } }
I20260928 21:07:12.747116 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:12.747375 30857 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dfcf53185d1c450d920bcbe0735776d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf53185d1c450d920bcbe0735776d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37743 } } }
I20260928 21:07:12.747475 30857 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:12.747493 30860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:12.751011 30860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:12.751745 30860 catalog_manager.cc:1259] Loaded cluster ID: 47f574c3b5b44bb29ec97122e93e0592
I20260928 21:07:12.751822 30860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:12.753507 30860 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:12.754177 30860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfcf53185d1c450d920bcbe0735776d3: Loaded TSK: 0
I20260928 21:07:12.755153 30860 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:07:12.836741 30854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:12.837062 30854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:12.837138 30854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:12.841502 30854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:12.841609 30854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:12.841708 30854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:12.847571 30854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.65.keytab
--principal=oryx/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:34305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=43367
--tserver_master_addrs=127.21.140.126:37743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:12.848801 30854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:12.849947 30854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:12.861351 30879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.861393 30878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:12.861711 30881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:12.861745 30854 server_base.cc:1034] running on GCE node
I20260928 21:07:12.862219 30854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:12.862885 30854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:12.864051 30854 hybrid_clock.cc:648] HybridClock initialized: now 1790629632864022 us; error 52 us; skew 500 ppm
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629632, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:12.867024 30854 init.cc:377] Logged in from keytab as oryx/127.21.140.65@KRBTEST.COM (short username oryx)
I20260928 21:07:12.867992 30854 webserver.cc:460] Webserver started at http://127.21.140.65:43367/ using document root <none> and password file <none>
I20260928 21:07:12.868572 30854 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:12.868649 30854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:12.872884 30854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:12.875202 30888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.876781 30854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:12.876928 30854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "ed5bb1138301410fa70103417ada2b2e"
format_stamp: "Formatted at 2026-09-28 21:07:11 on dist-test-slave-d5wc"
server_key: "5496666436496f9f664b8bb6bf3dde0c"
server_key_iv: "5c63fc1e144a55fbb7add97bd83e8822"
server_key_version: "encryptionkey@0"
I20260928 21:07:12.877502 30854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:12.892608 30854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:12.896041 30854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:12.896260 30854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:12.896865 30854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:12.897751 30854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:12.897822 30854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.897893 30854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:12.897938 30854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:12.907265 30854 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:34305
I20260928 21:07:12.907341 31001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:34305 every 8 connection(s)
I20260928 21:07:12.908298 30854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:12.910544 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 30854
I20260928 21:07:12.911062 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:42345
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:37743
--webserver_port=34395
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:12.923600 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.910571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:38855)
0928 21:07:12.911121 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:12.911181 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:12.912852 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:12.914179 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:12.914185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:12.914542 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:12.915053 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.915077 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.917168 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.917170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.918147 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.918157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.919311 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.919314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.919451 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.920076 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:12.920092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:12.922822 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
W20260928 21:07:12.923959 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:12.910437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:38855 (local address 127.21.140.126:37743)
0928 21:07:12.911027 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:12.911047 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:12.913103 (+ 2056us) server_negotiation.cc:407] Connection header received
0928 21:07:12.913317 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:12.913325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:12.913672 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:12.914047 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:12.915280 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.917022 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.918271 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.919137 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.919341 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.923116 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:38855: BlockingRecv error: recv got EOF from 127.21.140.65:38855 (error 108)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":111,"threads_started":1}
W20260928 21:07:12.924155 31002 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:37743 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.140.126:37743: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:12.930428 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.924672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:44729)
0928 21:07:12.924924 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:12.924936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:12.925015 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:12.925288 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:12.925291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:12.925434 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:12.925529 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.925534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.926581 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.926583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.927120 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.927127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.927710 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.927713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.927791 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.928366 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:12.928379 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:12.930317 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20260928 21:07:12.930603 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:12.924791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:44729 (local address 127.21.140.126:37743)
0928 21:07:12.924930 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:12.924934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:12.924977 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:07:12.925102 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:12.925105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:12.925158 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:12.925246 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:12.925634 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.926478 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.927225 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.927572 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.927675 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.930490 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:44729: BlockingRecv error: recv got EOF from 127.21.140.65:44729 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:12 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:12.936681 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:12.931108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:45507)
0928 21:07:12.931356 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:12.931375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:12.931449 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:12.931690 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:12.931692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:12.931816 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:12.931896 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.931902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.932884 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.932888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.933421 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:12.933429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.934118 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.934121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:12.934184 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.934678 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:12.934689 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:12.936579 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190}
W20260928 21:07:12.936880 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:12.931244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:45507 (local address 127.21.140.126:37743)
0928 21:07:12.931369 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:12.931374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:12.931434 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:07:12.931523 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:12.931526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:12.931580 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:12.931662 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:12.932002 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.932764 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.933532 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:12.933891 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:12.934022 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:12.936761 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:45507: BlockingRecv error: recv got EOF from 127.21.140.65:45507 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:07:12.937037 31002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:07:13.033311 31006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:13.033648 31006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:13.033754 31006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:13.039305 31006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:13.039443 31006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:13.039557 31006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:13.046909 31006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.66.keytab
--principal=oryx/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:42345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=34395
--tserver_master_addrs=127.21.140.126:37743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:13.048588 31006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:13.050037 31006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:13.061401 31012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:13.061429 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:13.061710 31006 server_base.cc:1034] running on GCE node
W20260928 21:07:13.061426 31014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:13.062371 31006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:13.062983 31006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:13.064185 31006 hybrid_clock.cc:648] HybridClock initialized: now 1790629633064167 us; error 32 us; skew 500 ppm
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629633, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:13.067358 31006 init.cc:377] Logged in from keytab as oryx/127.21.140.66@KRBTEST.COM (short username oryx)
I20260928 21:07:13.068418 31006 webserver.cc:460] Webserver started at http://127.21.140.66:34395/ using document root <none> and password file <none>
I20260928 21:07:13.069118 31006 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:13.069237 31006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:13.073096 31006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:13.075268 31021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:13.076479 31006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:07:13.076623 31006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "ab80259eeb634d55b6015e96e89f80b7"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "d137d01a34782cd2e2903b50d422cc36"
server_key_iv: "773550f101db560ea85dc663a7e17585"
server_key_version: "encryptionkey@0"
I20260928 21:07:13.077251 31006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:13.098866 31006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:13.102218 31006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:13.102385 31006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:13.102952 31006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:13.103749 31006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:13.103798 31006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:13.103840 31006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:13.103855 31006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:13.113260 31006 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:42345
I20260928 21:07:13.113286 31134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:42345 every 8 connection(s)
I20260928 21:07:13.114302 31006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:13.118702 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31006
I20260928 21:07:13.119203 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:41903
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:37743
--webserver_port=42511
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:07:13.127158 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.116257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:32973 (local address 127.21.140.126:37743)
0928 21:07:13.116395 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.116399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.117321 (+ 922us) server_negotiation.cc:407] Connection header received
0928 21:07:13.118699 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.118702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.118753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.118889 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.120571 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.121468 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.122700 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.123124 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.123254 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.126987 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:32973: BlockingRecv error: recv got EOF from 127.21.140.66:32973 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:07:13.127612 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.116656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:32973)
0928 21:07:13.117155 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.117197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.118455 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.119095 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.119107 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.119647 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.120369 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.120392 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.121612 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.121615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.122563 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.122573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.123300 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.123303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.123460 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.124139 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.124163 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.126777 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":105,"threads_started":1}
W20260928 21:07:13.128221 31135 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:37743 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.140.126:37743: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:13.134382 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.128707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:57645)
0928 21:07:13.128969 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.128981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.129044 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.129299 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.129300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.129458 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.129547 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.129551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.130694 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.130697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.131243 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.131252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.131844 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.131848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.131930 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.132421 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.132432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.134276 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199}
W20260928 21:07:13.134747 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.128850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:57645 (local address 127.21.140.126:37743)
0928 21:07:13.129000 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.129006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.129024 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:07:13.129119 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.129122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.129179 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.129277 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.129663 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.130572 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.131361 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.131716 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.131818 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.134444 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:57645: BlockingRecv error: recv got EOF from 127.21.140.66:57645 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:13.140359 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.135057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:56305)
0928 21:07:13.135262 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.135275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.135362 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.135584 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.135587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.135746 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.135831 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.135837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.136518 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.136522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.137399 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.137420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.137826 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.137830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.137915 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.138374 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.138388 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.140242 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:07:13.140645 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.135111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:56305 (local address 127.21.140.126:37743)
0928 21:07:13.135203 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.135207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.135376 (+ 169us) server_negotiation.cc:407] Connection header received
0928 21:07:13.135444 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.135446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.135482 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.135548 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.135938 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.136422 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.137528 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.137725 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.137802 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.140387 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:56305: BlockingRecv error: recv got EOF from 127.21.140.66:56305 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 21:07:13.140748 31135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:07:13.236763 31138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:13.237077 31138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:13.237149 31138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:13.241251 31138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:13.241348 31138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:13.241434 31138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:13.246948 31138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/krb5kdc/oryx_127.21.140.67.keytab
--principal=oryx/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:41903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=42511
--tserver_master_addrs=127.21.140.126:37743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:13.248261 31138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:13.249352 31138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:13.259164 31143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:13.259166 31144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:13.259204 31146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:13.259212 31138 server_base.cc:1034] running on GCE node
I20260928 21:07:13.259775 31138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:13.260483 31138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:13.261652 31138 hybrid_clock.cc:648] HybridClock initialized: now 1790629633261636 us; error 30 us; skew 500 ppm
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629633, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:13.264565 31138 init.cc:377] Logged in from keytab as oryx/127.21.140.67@KRBTEST.COM (short username oryx)
I20260928 21:07:13.265548 31138 webserver.cc:460] Webserver started at http://127.21.140.67:42511/ using document root <none> and password file <none>
I20260928 21:07:13.266150 31138 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:13.266230 31138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:13.270092 31138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 21:07:13.272138 31153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:13.273360 31138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:13.273505 31138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "b4a9d1c80bd6432d916c5ff044b01cbb"
format_stamp: "Formatted at 2026-09-28 21:07:12 on dist-test-slave-d5wc"
server_key: "df6bfdd6e5bda66dea5b7bd1bc190869"
server_key_iv: "136865b541c6da23dfe13d2fadfb2c89"
server_key_version: "encryptionkey@0"
I20260928 21:07:13.274082 31138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:13.298367 31138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:13.301786 31138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:13.302034 31138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:13.302685 31138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:13.303529 31138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:13.303574 31138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:13.303615 31138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:13.303671 31138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:13.312731 31138 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:41903
I20260928 21:07:13.312753 31266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:41903 every 8 connection(s)
I20260928 21:07:13.313724 31138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:07:13.316581 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31138
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:07:13.325708 31004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.315569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50025 (local address 127.21.140.126:37743)
0928 21:07:13.315749 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.315754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.316596 (+ 842us) server_negotiation.cc:407] Connection header received
0928 21:07:13.318371 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.318374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.318416 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.318490 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.319978 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.320459 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.321605 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.321819 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.321933 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.325603 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:50025: BlockingRecv error: recv got EOF from 127.21.140.67:50025 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:07:13.326541 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.315991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:50025)
0928 21:07:13.316440 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.316483 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.318146 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.318662 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.318670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.319173 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.319832 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.319855 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.320588 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.320591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.321472 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.321481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.322008 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.322013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.322177 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.322703 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.322735 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.325458 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":97,"threads_started":1}
W20260928 21:07:13.327085 31267 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:37743 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.140.126:37743: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
I20260928 21:07:13.327406 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.317580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.21.140.126:37743)
0928 21:07:13.317840 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.317845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.317860 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:07:13.317905 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.317908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.317956 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.318089 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.318844 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.319640 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.320604 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.320834 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.320969 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.322523 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:13.322529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:13.322542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:13.322598 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:13.324594 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:13.325101 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:13.325106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:13.325111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:13.325232 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:13.325564 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:13.325567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:13.325569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:13.326069 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:13.326280 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:13.326579 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 21:07:13.327236 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:13.333425 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.327561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:32779)
0928 21:07:13.327679 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.327694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.327767 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.328707 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.328710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.328838 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.328977 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.328983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.329805 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.329810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.330556 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.330565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.331050 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.331057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.331134 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.331546 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.331557 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.333276 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:07:13.333508 31271 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.328380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:32779 (local address 127.21.140.126:37743)
0928 21:07:13.328505 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.328509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.328523 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:13.328555 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.328557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.328595 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.328674 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.329097 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.329701 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.330676 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.330891 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.330989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.333391 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:32779: BlockingRecv error: recv got EOF from 127.21.140.67:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:13.338809 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.334122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:41283)
0928 21:07:13.334240 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.334257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.334335 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.334512 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.334514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.334630 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.334693 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.334697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.335417 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.335419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.336013 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.336022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.336465 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.336469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.336530 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.336935 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.336946 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.338664 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:07:13.338889 31271 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.334191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:41283 (local address 127.21.140.126:37743)
0928 21:07:13.334317 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.334320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.334332 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:07:13.334394 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.334396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.334426 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.334488 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.334811 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.335292 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.336134 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.336335 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.336443 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.338774 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:41283: BlockingRecv error: recv got EOF from 127.21.140.67:41283 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 21:07:13.339082 31267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:13 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:13.943445 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:13.937736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:33219)
0928 21:07:13.938063 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:13.938078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:13.938172 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:13.938431 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:13.938433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:13.938640 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:13.938754 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.938759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.939548 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.939551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.940126 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:13.940134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.940582 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.940585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:13.940656 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.941261 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:13.941273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:13.943269 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
W20260928 21:07:13.943666 31273 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:13.937812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:33219 (local address 127.21.140.126:37743)
0928 21:07:13.938115 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:13.938118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:13.938133 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:07:13.938271 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:13.938274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:13.938318 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:13.938410 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:13.938872 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.939423 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.940237 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:13.940476 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:13.940558 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:13.943392 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:33219: BlockingRecv error: recv got EOF from 127.21.140.65:33219 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
Sep 28 21:07:14 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:14 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:14.147374 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:14.141485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:34195)
0928 21:07:14.141769 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:14.141783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:14.141886 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:14.142156 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:14.142159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:14.142344 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:14.142465 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:14.142471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.143238 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.143241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:14.143875 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:14.143883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.144544 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.144549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:14.144654 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:14.145259 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:14.145272 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:14.147250 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
W20260928 21:07:14.147557 31273 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:14.141570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:34195 (local address 127.21.140.126:37743)
0928 21:07:14.141754 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:14.141757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:14.141858 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:07:14.141987 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:14.141989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:14.142035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:14.142109 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:14.142605 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.143137 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.144058 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.144408 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.144530 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:14.147389 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:34195: BlockingRecv error: recv got EOF from 127.21.140.66:34195 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 21:07:14 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:14 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:07:14.346030 31273 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:14.339845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:35803 (local address 127.21.140.126:37743)
0928 21:07:14.340107 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:14.340112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:14.340263 (+ 151us) server_negotiation.cc:407] Connection header received
0928 21:07:14.340430 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:14.340434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:14.340479 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:14.340565 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:14.341044 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.341596 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.342527 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.342780 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.342887 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:14.345901 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:35803: BlockingRecv error: recv got EOF from 127.21.140.67:35803 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
I20260928 21:07:14.346030 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:14.339701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:35803)
0928 21:07:14.340175 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:14.340188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:14.340320 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:14.340585 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:14.340588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:14.340790 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:14.340925 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:14.340931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.341713 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.341718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:14.342361 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:14.342370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.342910 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.342915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:14.343021 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:14.343669 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:14.343684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:14.345768 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":102,"threads_started":1}
Sep 28 21:07:14 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:14 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:14.949786 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:14.944234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:36235)
0928 21:07:14.944493 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:14.944511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:14.944617 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:14.944849 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:14.944851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:14.945061 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:14.945157 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:14.945163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.945899 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.945901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:14.946521 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:14.946530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.947041 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.947044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:14.947155 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:14.947621 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:14.947632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:14.949659 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
W20260928 21:07:14.950037 31277 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:14.944349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:36235 (local address 127.21.140.126:37743)
0928 21:07:14.944607 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:14.944610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:14.944623 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:07:14.944698 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:14.944700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:14.944744 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:14.944812 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:14.945262 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.945800 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.946633 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:14.946943 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:14.947026 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:14.949862 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:36235: BlockingRecv error: recv got EOF from 127.21.140.65:36235 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
Sep 28 21:07:15 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:15 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:15.154426 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:15.148198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:48545)
0928 21:07:15.148698 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:15.148712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:15.148818 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:15.149070 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:15.149072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:15.149256 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:15.149350 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:15.149355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.150199 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.150202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:15.150773 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:15.150782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.151242 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.151245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:15.151303 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:15.151801 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:15.151812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:15.154232 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":144,"threads_started":1}
W20260928 21:07:15.154497 31277 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:15.148381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:48545 (local address 127.21.140.126:37743)
0928 21:07:15.148594 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:15.148598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:15.148795 (+ 197us) server_negotiation.cc:407] Connection header received
0928 21:07:15.148906 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:15.148910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:15.148957 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:15.149035 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:15.149478 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.150095 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.150891 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.151141 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.151231 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:15.154394 (+ 3163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:48545: BlockingRecv error: recv got EOF from 127.21.140.66:48545 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 21:07:15 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:15 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:15.358719 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:15.346951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:50405)
0928 21:07:15.347235 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:15.347252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:15.347434 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:15.347772 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:15.347775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:15.348035 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:15.348138 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:15.348143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.348974 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.348980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:15.349558 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:15.349566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.350100 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.350104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:15.350190 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:15.356599 (+ 6409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:15.356613 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:15.358605 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
W20260928 21:07:15.359048 31277 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:15.347080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50405 (local address 127.21.140.126:37743)
0928 21:07:15.347228 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:15.347233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:15.347337 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:07:15.347559 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:15.347562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:15.347658 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:15.347752 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:15.348258 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.348866 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.349683 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.349926 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.350095 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:15.358764 (+ 8669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:50405: BlockingRecv error: recv got EOF from 127.21.140.67:50405 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 21:07:15 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:15 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:15.956871 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:15.950620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:54751)
0928 21:07:15.950970 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:15.950985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:15.951097 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:15.951486 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:15.951489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:15.951782 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:15.951894 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:15.951901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.952977 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.952980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:15.953677 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:15.953693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.954133 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.954136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:15.954199 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:15.954775 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:15.954786 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:15.956742 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":69,"threads_started":1}
W20260928 21:07:15.957022 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:15.950690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:54751 (local address 127.21.140.126:37743)
0928 21:07:15.950983 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:15.950986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:15.951065 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:07:15.951224 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:15.951227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:15.951273 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:15.951453 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:15.952047 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.952856 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.953790 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:15.954037 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:15.954115 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:15.956885 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:54751: BlockingRecv error: recv got EOF from 127.21.140.65:54751 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
Sep 28 21:07:16 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:16 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:16.160534 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:16.155283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:36469)
0928 21:07:16.155581 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:16.155594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:16.155691 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:16.155920 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:16.155923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:16.156101 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:16.156192 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:16.156197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.156944 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.156948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:16.157498 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:16.157506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.157996 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.157999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:16.158063 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:16.158547 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:16.158559 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:16.160412 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
W20260928 21:07:16.160789 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:16.155385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:36469 (local address 127.21.140.126:37743)
0928 21:07:16.155586 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:16.155597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:16.155667 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:07:16.155770 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:16.155773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:16.155816 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:16.155884 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:16.156304 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.156838 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.157628 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.157847 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.157948 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:16.160635 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:36469: BlockingRecv error: recv got EOF from 127.21.140.66:36469 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 28 21:07:16 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:16 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:16.366129 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:16.359624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:39167)
0928 21:07:16.359971 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:16.359995 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:16.360156 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:16.360438 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:16.360441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:16.360674 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:16.360799 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:16.360811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.362177 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.362180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:16.362755 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:16.362763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.363276 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.363279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:16.363349 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:16.363857 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:16.363869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:16.365761 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
W20260928 21:07:16.366144 31281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:16.359751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:39167 (local address 127.21.140.126:37743)
0928 21:07:16.359937 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:16.359941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:16.360075 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:07:16.360272 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:16.360279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:16.360327 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:16.360420 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:16.360973 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.362051 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.362922 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.363144 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.363242 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:16.365997 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:39167: BlockingRecv error: recv got EOF from 127.21.140.67:39167 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 21:07:16 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:16 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:16.963443 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:16.957765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:58151)
0928 21:07:16.958096 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:16.958115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:16.958242 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:16.958554 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:16.958557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:16.958768 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:16.958868 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:16.958874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.959722 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.959725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:16.960282 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:16.960290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.960746 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.960748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:16.960814 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:16.961325 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:16.961336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:16.963323 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
W20260928 21:07:16.963663 31285 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:16.957850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:58151 (local address 127.21.140.126:37743)
0928 21:07:16.958227 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:16.958230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:16.958244 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:16.958355 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:16.958359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:16.958414 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:16.958532 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:16.959027 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.959581 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.960397 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:16.960635 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:16.960722 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:16.963505 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:58151: BlockingRecv error: recv got EOF from 127.21.140.65:58151 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":181,"threads_started":1}
Sep 28 21:07:17 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:17 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:17.167050 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:17.161473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:36623)
0928 21:07:17.161784 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:17.161796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:17.161939 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:17.162229 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:17.162232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:17.162447 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:17.162542 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:17.162548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.163336 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.163341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:17.163891 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:17.163899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.164360 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.164363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:17.164449 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:17.164990 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:17.165001 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:17.166895 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
W20260928 21:07:17.167215 31285 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:17.161558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:36623 (local address 127.21.140.126:37743)
0928 21:07:17.161732 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:17.161737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:17.161872 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:07:17.162058 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:17.162062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:17.162118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:17.162206 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:17.162666 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.163220 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.164019 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.164244 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.164356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:17.167058 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:36623: BlockingRecv error: recv got EOF from 127.21.140.66:36623 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 21:07:17 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:17 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:17.372337 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:17.366910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:50445)
0928 21:07:17.367211 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:17.367226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:17.367348 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:17.367636 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:17.367638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:17.367818 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:17.367922 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:17.367928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.368672 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.368675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:17.369227 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:17.369235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.369721 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.369724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:17.369806 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:17.370401 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:17.370424 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:17.372224 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
W20260928 21:07:17.372496 31285 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:17.366988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50445 (local address 127.21.140.126:37743)
0928 21:07:17.367142 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:17.367147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:17.367288 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:07:17.367468 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:17.367470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:17.367519 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:17.367587 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:17.368032 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.368580 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.369357 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.369581 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.369672 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:17.372368 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:50445: BlockingRecv error: recv got EOF from 127.21.140.67:50445 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 21:07:17 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:17 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:17.970283 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:17.964275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:42285)
0928 21:07:17.964668 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:17.964706 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:17.964821 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:17.965121 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:17.965124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:17.965391 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:17.965526 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:17.965534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.966440 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.966443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:17.967002 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:17.967009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.967435 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.967438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:17.967502 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:17.968259 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:17.968271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:17.970168 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":102,"threads_started":1}
W20260928 21:07:17.970497 31289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:17.964429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:42285 (local address 127.21.140.126:37743)
0928 21:07:17.964719 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:17.964723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:17.964767 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:07:17.964939 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:17.964942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:17.964996 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:17.965093 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:17.965654 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.966340 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.967110 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:17.967330 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:17.967411 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:17.970353 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:42285: BlockingRecv error: recv got EOF from 127.21.140.65:42285 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
Sep 28 21:07:18 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:18 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:18.173765 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:18.167857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:60639)
0928 21:07:18.168155 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:18.168168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:18.168252 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:18.168535 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:18.168538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:18.168768 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:18.168910 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:18.168916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.169752 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.169755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:18.170477 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:18.170487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.170973 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.170976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:18.171056 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:18.171561 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:18.171573 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:18.173646 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":115,"threads_started":1}
W20260928 21:07:18.173959 31289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:18.168054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:60639 (local address 127.21.140.126:37743)
0928 21:07:18.168196 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:18.168199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:18.168213 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:18.168337 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:18.168339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:18.168388 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:18.168462 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:18.169057 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.169627 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.170616 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.170841 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.170932 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:18.173815 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:60639: BlockingRecv error: recv got EOF from 127.21.140.66:60639 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 21:07:18 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:18 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:18.379153 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:18.373256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:52065)
0928 21:07:18.373525 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:18.373541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:18.373639 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:18.373892 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:18.373894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:18.374128 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:18.374223 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:18.374228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.374962 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.374965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:18.375577 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:18.375584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.376173 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.376176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:18.376248 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:18.376802 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:18.376815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:18.379021 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":66,"threads_started":1}
W20260928 21:07:18.379315 31289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:18.373394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:52065 (local address 127.21.140.126:37743)
0928 21:07:18.373529 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:18.373533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:18.373596 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:07:18.373735 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:18.373737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:18.373784 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:18.373849 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:18.374323 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.374835 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.375749 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.376063 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.376144 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:18.379152 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:52065: BlockingRecv error: recv got EOF from 127.21.140.67:52065 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 21:07:18 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:18 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:18.976727 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:18.971133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:43517)
0928 21:07:18.971423 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:18.971436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:18.971535 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:18.971787 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:18.971789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:18.971987 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:18.972084 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:18.972090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.972911 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.972915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:18.973517 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:18.973525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.974050 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.974054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:18.974143 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:18.974717 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:18.974729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:18.976552 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
W20260928 21:07:18.976799 31293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:18.971281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:43517 (local address 127.21.140.126:37743)
0928 21:07:18.971530 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:18.971534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:18.971548 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:18.971610 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:18.971613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:18.971654 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:18.971739 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:18.972205 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.972784 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.973633 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:18.973895 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:18.974037 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:18.976678 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:43517: BlockingRecv error: recv got EOF from 127.21.140.65:43517 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
Sep 28 21:07:19 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:19 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:19.180979 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:19.174855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:60991)
0928 21:07:19.175201 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:19.175217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:19.175329 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:19.175670 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:19.175673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:19.176192 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:19.176323 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:19.176329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.177096 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.177099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:19.177657 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:19.177665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.178278 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.178281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:19.178347 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:19.178830 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:19.178841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:19.180788 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
W20260928 21:07:19.181075 31293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:19.175022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:60991 (local address 127.21.140.126:37743)
0928 21:07:19.175191 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:19.175195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:19.175301 (+ 106us) server_negotiation.cc:407] Connection header received
0928 21:07:19.175449 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:19.175455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:19.175520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:19.175625 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:19.176435 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.176980 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.177832 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.178162 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.178247 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:19.180962 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:60991: BlockingRecv error: recv got EOF from 127.21.140.66:60991 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 21:07:19 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:19 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:19.386565 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:19.379941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:59529)
0928 21:07:19.380185 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:19.380200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:19.380305 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:19.380618 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:19.380621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:19.380870 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:19.381089 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:19.381097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.382090 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.382094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:19.383071 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:19.383081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.383533 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.383536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:19.383600 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:19.384106 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:19.384118 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:19.386394 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
W20260928 21:07:19.386623 31293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:19.380050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:59529 (local address 127.21.140.126:37743)
0928 21:07:19.380185 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:19.380195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:19.380260 (+ 65us) server_negotiation.cc:407] Connection header received
0928 21:07:19.380428 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:19.380431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:19.380509 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:19.380613 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:19.381226 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.381935 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.383190 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.383418 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.383503 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:19.386516 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:59529: BlockingRecv error: recv got EOF from 127.21.140.67:59529 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 21:07:19 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:19 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:19.983811 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:19.977633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:51479)
0928 21:07:19.978050 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:19.978065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:19.978164 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:19.978417 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:19.978419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:19.978601 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:19.978692 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:19.978697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.979568 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.979571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:19.980148 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:19.980156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.980647 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.980650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:19.980776 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:19.981386 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:19.981397 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:19.983699 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
W20260928 21:07:19.984176 31297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:19.977716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:51479 (local address 127.21.140.126:37743)
0928 21:07:19.978075 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:19.978078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:19.978134 (+ 56us) server_negotiation.cc:407] Connection header received
0928 21:07:19.978245 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:19.978248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:19.978293 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:19.978371 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:19.978817 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.979466 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.980257 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:19.980550 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:19.980632 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:19.983891 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:51479: BlockingRecv error: recv got EOF from 127.21.140.65:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":131,"threads_started":1}
Sep 28 21:07:20 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:20 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:20.187453 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:20.181774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:45349)
0928 21:07:20.182106 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:20.182121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:20.182227 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:20.182488 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:20.182490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:20.182676 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:20.182768 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:20.182773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.183608 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.183611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:20.184483 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:20.184494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.184936 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.184939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:20.184999 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:20.185494 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:20.185505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:20.187337 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
W20260928 21:07:20.187601 31297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:20.181871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:45349 (local address 127.21.140.126:37743)
0928 21:07:20.182090 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:20.182094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:20.182198 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:07:20.182294 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:20.182296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:20.182356 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:20.182425 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:20.182864 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.183484 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.184601 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.184836 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.184951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:20.187483 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:45349: BlockingRecv error: recv got EOF from 127.21.140.66:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 28 21:07:20 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:20 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:20.398016 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:20.387499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:39361)
0928 21:07:20.387796 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:20.387809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:20.387951 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:20.388305 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:20.388308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:20.388501 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:20.388599 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:20.388604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.389388 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.389394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:20.389955 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:20.390034 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.390492 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.390495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:20.390592 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:20.395576 (+ 4984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:20.395589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:20.397831 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
W20260928 21:07:20.398263 31297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:20.387531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:39361 (local address 127.21.140.126:37743)
0928 21:07:20.387691 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:20.387696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:20.387895 (+ 199us) server_negotiation.cc:407] Connection header received
0928 21:07:20.388084 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:20.388092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:20.388168 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:20.388248 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:20.388733 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.389273 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.390141 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.390374 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.390489 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:20.398124 (+ 7635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:39361: BlockingRecv error: recv got EOF from 127.21.140.67:39361 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 21:07:20 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:20 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:20.991678 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:20.984784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:32889)
0928 21:07:20.985142 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:20.985157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:20.985281 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:20.985543 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:20.985545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:20.985781 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:20.985905 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:20.985912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.986818 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.986821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:20.987464 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:20.987475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.987977 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.987980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:20.988050 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:20.988720 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:20.988735 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:20.991434 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":129,"threads_started":1}
W20260928 21:07:20.991904 31301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:20.984999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:32889 (local address 127.21.140.126:37743)
0928 21:07:20.985274 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:20.985277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:20.985295 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:07:20.985354 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:20.985356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:20.985406 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:20.985500 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:20.986089 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.986706 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.987577 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:20.987860 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:20.987944 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:20.991718 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:32889: BlockingRecv error: recv got EOF from 127.21.140.65:32889 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":131,"threads_started":1}
Sep 28 21:07:21 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:21 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:21.195822 31302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:21.188671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:53167)
0928 21:07:21.189014 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:21.189034 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:21.189209 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:21.189574 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:21.189577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:21.189823 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:21.189952 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:21.189958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.191102 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.191105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:21.191746 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:21.191756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.192318 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.192321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:21.192392 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:21.193131 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:21.193149 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:21.195649 (+ 2500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
W20260928 21:07:21.196007 31301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:21.188727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:53167 (local address 127.21.140.126:37743)
0928 21:07:21.188925 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:21.188929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:21.189137 (+ 208us) server_negotiation.cc:407] Connection header received
0928 21:07:21.189320 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:21.189323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:21.189389 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:21.189480 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:21.190220 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.190989 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.191880 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.192167 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.192305 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:21.195834 (+ 3529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:53167: BlockingRecv error: recv got EOF from 127.21.140.66:53167 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 21:07:21 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:21 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:21.406692 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:21.399061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:38673)
0928 21:07:21.399416 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:21.399434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:21.399590 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:21.399931 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:21.399934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:21.400158 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:21.400300 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:21.400306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.401143 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.401146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:21.401768 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:21.401777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.402554 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.402558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:21.402645 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:21.403714 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:21.403735 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:21.406502 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
W20260928 21:07:21.407032 31301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:21.399264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:38673 (local address 127.21.140.126:37743)
0928 21:07:21.399455 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:21.399459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:21.399530 (+ 71us) server_negotiation.cc:407] Connection header received
0928 21:07:21.399708 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:21.399712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:21.399775 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:21.399863 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:21.400426 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.401037 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.401893 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.402252 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.402652 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:21.406873 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:38673: BlockingRecv error: recv got EOF from 127.21.140.67:38673 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 21:07:21 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:21 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:21.999804 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:21.993069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:34847)
0928 21:07:21.993386 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:21.993400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:21.993498 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:21.993744 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:21.993746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:21.993948 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:21.994090 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:21.994096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.994887 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.994891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:21.995731 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:21.995741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.996407 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.996411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:21.996502 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:21.997138 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:21.997155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:21.999657 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
W20260928 21:07:21.999995 31305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:21.993174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:34847 (local address 127.21.140.126:37743)
0928 21:07:21.993465 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:21.993469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:21.993483 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:21.993572 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:21.993575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:21.993620 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:21.993705 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:21.994184 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.994764 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.995837 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:21.996233 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:21.996353 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:21.999812 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:34847: BlockingRecv error: recv got EOF from 127.21.140.65:34847 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
Sep 28 21:07:22 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:22 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:22.202935 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:22.196829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:51171)
0928 21:07:22.197209 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:22.197229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:22.197396 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:22.197702 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:22.197705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:22.197932 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:22.198069 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:22.198075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.198841 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.198845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:22.199456 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:22.199467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.199954 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.199957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:22.200049 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:22.200666 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:22.200687 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:22.202789 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
W20260928 21:07:22.203234 31305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:22.196901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:51171 (local address 127.21.140.126:37743)
0928 21:07:22.197058 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:22.197063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:22.197306 (+ 243us) server_negotiation.cc:407] Connection header received
0928 21:07:22.197525 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:22.197530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:22.197585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:22.197679 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:22.198196 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.198724 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.199586 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.199833 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.199950 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:22.202999 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:51171: BlockingRecv error: recv got EOF from 127.21.140.66:51171 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:07:22 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:22 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:22.413769 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:22.407767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:43865)
0928 21:07:22.408103 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:22.408126 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:22.408230 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:22.408492 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:22.408495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:22.408679 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:22.408787 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:22.408792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.409598 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.409600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:22.410282 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:22.410291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.410775 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.410779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:22.410865 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:22.411439 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:22.411451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:22.413581 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
W20260928 21:07:22.413841 31305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:22.407926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:43865 (local address 127.21.140.126:37743)
0928 21:07:22.408065 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:22.408072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:22.408220 (+ 148us) server_negotiation.cc:407] Connection header received
0928 21:07:22.408301 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:22.408304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:22.408354 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:22.408456 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:22.408948 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.409500 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.410415 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:22.410650 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:22.410775 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:22.413729 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:43865: BlockingRecv error: recv got EOF from 127.21.140.67:43865 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 21:07:23 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:23 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:23.006888 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:23.000692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:47939)
0928 21:07:23.001057 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:23.001073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:23.001194 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:23.001456 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:23.001459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:23.001669 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:23.001804 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:23.001813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.002654 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.002658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:23.003426 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:23.003436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.003910 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.003914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:23.004177 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:23.004658 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:23.004669 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:23.006711 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":86,"threads_started":1}
W20260928 21:07:23.007006 31309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:23.000793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:47939 (local address 127.21.140.126:37743)
0928 21:07:23.001090 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:23.001094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:23.001168 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:07:23.001282 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:23.001286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:23.001341 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:23.001439 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:23.001940 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.002525 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.003542 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.003778 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.003904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:23.006857 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:47939: BlockingRecv error: recv got EOF from 127.21.140.65:47939 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
Sep 28 21:07:23 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:23 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:23.209570 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:23.203740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:52689)
0928 21:07:23.204074 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:23.204091 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:23.204241 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:23.204523 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:23.204526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:23.204736 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:23.204825 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:23.204830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.205645 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.205650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:23.206397 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:23.206407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.206952 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.206955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:23.207044 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:23.207614 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:23.207626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:23.209453 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
W20260928 21:07:23.209887 31309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:23.203921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:52689 (local address 127.21.140.126:37743)
0928 21:07:23.204077 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:23.204085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:23.204181 (+ 96us) server_negotiation.cc:407] Connection header received
0928 21:07:23.204353 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:23.204356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:23.204411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:23.204501 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:23.204982 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.205526 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.206539 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.206840 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.206938 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:23.209631 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:52689: BlockingRecv error: recv got EOF from 127.21.140.66:52689 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:07:23 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:23 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:23.420459 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:23.414708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:42775)
0928 21:07:23.414991 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:23.415006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:23.415089 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:23.415359 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:23.415362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:23.415573 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:23.415670 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:23.415676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.416551 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.416554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:23.417147 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:23.417156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.417642 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.417645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:23.417713 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:23.418387 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:23.418399 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:23.420335 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
W20260928 21:07:23.420672 31309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:23.414965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:42775 (local address 127.21.140.126:37743)
0928 21:07:23.415110 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:23.415113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:23.415126 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:07:23.415164 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:23.415167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:23.415208 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:23.415283 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:23.415822 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.416406 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.417270 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:23.417519 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:23.417616 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:23.420540 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:42775: BlockingRecv error: recv got EOF from 127.21.140.67:42775 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 21:07:24 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:24 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:24.015352 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:24.008039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:59715)
0928 21:07:24.008422 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:24.008445 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:24.008605 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:24.008956 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:24.008959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:24.009194 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:24.009332 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:24.009338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.010298 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.010301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:24.010917 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:24.010929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.011568 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.011571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:24.011680 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:24.012639 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:24.012667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:24.015155 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
W20260928 21:07:24.015611 31313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:24.008174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:59715 (local address 127.21.140.126:37743)
0928 21:07:24.008495 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:24.008503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:24.008524 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:07:24.008744 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:24.008748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:24.008819 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:24.008914 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:24.009464 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.010170 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.011079 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.011402 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.011509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:24.015424 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:59715: BlockingRecv error: recv got EOF from 127.21.140.65:59715 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
Sep 28 21:07:24 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:24 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:24.217619 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:24.210703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:56003)
0928 21:07:24.211102 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:24.211119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:24.211299 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:24.211630 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:24.211633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:24.211904 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:24.212025 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:24.212031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.212907 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.212912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:24.213528 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:24.213538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.214132 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.214136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:24.214241 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:24.214974 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:24.214989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:24.217411 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
W20260928 21:07:24.217763 31313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:24.210784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:56003 (local address 127.21.140.126:37743)
0928 21:07:24.211034 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:24.211040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:24.211212 (+ 172us) server_negotiation.cc:407] Connection header received
0928 21:07:24.211418 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:24.211422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:24.211494 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:24.211601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:24.212166 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.212771 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.213669 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.213932 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.214109 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:24.217630 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:56003: BlockingRecv error: recv got EOF from 127.21.140.66:56003 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 21:07:24 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:24 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:24.428979 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:24.421645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:33437)
0928 21:07:24.422075 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:24.422099 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:24.422273 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:24.422638 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:24.422642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:24.422925 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:24.423046 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:24.423054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.424025 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.424031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:24.424725 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:24.424736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.425324 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.425328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:24.425434 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:24.426285 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:24.426301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:24.428785 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
W20260928 21:07:24.429283 31313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:24.421943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:33437 (local address 127.21.140.126:37743)
0928 21:07:24.422219 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:24.422225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:24.422245 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:07:24.422418 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:24.422422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:24.422494 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:24.422608 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:24.423202 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.423879 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.424890 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:24.425174 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:24.425312 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:24.429060 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:33437: BlockingRecv error: recv got EOF from 127.21.140.67:33437 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 28 21:07:25 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:25 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:25.022387 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:25.016419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:55479)
0928 21:07:25.016787 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:25.016802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:25.016913 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:25.017208 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:25.017211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:25.017429 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:25.017542 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:25.017548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.018470 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.018473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:25.019061 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:25.019070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.019540 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.019543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:25.019613 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:25.020141 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:25.020153 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:25.022201 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":96,"threads_started":1}
W20260928 21:07:25.022469 31317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:25.016571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:55479 (local address 127.21.140.126:37743)
0928 21:07:25.016840 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:25.016843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:25.016858 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:07:25.017035 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:25.017037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:25.017086 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:25.017168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:25.017690 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.018358 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.019181 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.019404 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.019500 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:25.022332 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:55479: BlockingRecv error: recv got EOF from 127.21.140.65:55479 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":65,"threads_started":1}
Sep 28 21:07:25 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:25 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:25.224483 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:25.218641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:46613)
0928 21:07:25.218930 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:25.218954 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:25.219058 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:25.219398 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:25.219401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:25.219611 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:25.219729 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:25.219735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.220570 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.220573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:25.221183 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:25.221192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.221661 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.221665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:25.221733 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:25.222259 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:25.222271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:25.224306 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
W20260928 21:07:25.224723 31317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:25.218776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:46613 (local address 127.21.140.126:37743)
0928 21:07:25.218917 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:25.218921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:25.219027 (+ 106us) server_negotiation.cc:407] Connection header received
0928 21:07:25.219181 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:25.219184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:25.219232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:25.219312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:25.219916 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.220463 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.221305 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.221529 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.221613 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:25.224472 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:46613: BlockingRecv error: recv got EOF from 127.21.140.66:46613 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 21:07:25 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:25 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:25.440964 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:25.430046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:51529)
0928 21:07:25.430354 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:25.430368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:25.430476 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:25.430743 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:25.430745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:25.430971 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:25.431077 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:25.431083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.432013 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.432016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:25.432605 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:25.432614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.433156 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.433159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:25.433224 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:25.438697 (+ 5473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:25.438710 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:25.440830 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":116,"threads_started":1}
W20260928 21:07:25.441274 31317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:25.430193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:51529 (local address 127.21.140.126:37743)
0928 21:07:25.430340 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:25.430351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:25.430429 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:07:25.430570 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:25.430573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:25.430621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:25.430725 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:25.431218 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.431748 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.432770 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:25.433034 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:25.433127 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:25.441032 (+ 7905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:51529: BlockingRecv error: recv got EOF from 127.21.140.67:51529 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 21:07:26 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:26 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:26.029484 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:26.023222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:44599)
0928 21:07:26.023540 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:26.023555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:26.023666 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:26.024078 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:26.024081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:26.024279 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:26.024378 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:26.024384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.025388 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.025391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:26.026251 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:26.026263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.026840 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.026843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:26.026944 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:26.027546 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:26.027567 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:26.029365 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":86,"threads_started":1}
W20260928 21:07:26.029659 31321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:26.023294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:44599 (local address 127.21.140.126:37743)
0928 21:07:26.023559 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:26.023563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:26.023635 (+ 72us) server_negotiation.cc:407] Connection header received
0928 21:07:26.023801 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:26.023804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:26.023856 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:26.024000 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:26.024521 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.025167 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.026406 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.026728 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.026809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:26.029516 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:44599: BlockingRecv error: recv got EOF from 127.21.140.65:44599 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
Sep 28 21:07:26 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:26 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:26.230981 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:26.225333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:34157)
0928 21:07:26.225617 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:26.225633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:26.225755 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:26.226023 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:26.226026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:26.226282 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:26.226384 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:26.226390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.227181 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.227186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:26.227769 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:26.227777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.228254 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.228257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:26.228347 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:26.228875 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:26.228887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:26.230845 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
W20260928 21:07:26.231174 31321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:26.225474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:34157 (local address 127.21.140.126:37743)
0928 21:07:26.225630 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:26.225637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:26.225706 (+ 69us) server_negotiation.cc:407] Connection header received
0928 21:07:26.225853 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:26.225856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:26.225907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:26.226007 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:26.226516 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.227060 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.227901 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.228133 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.228258 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:26.231038 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:34157: BlockingRecv error: recv got EOF from 127.21.140.66:34157 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 21:07:26 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:26 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:26.447525 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:26.441783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:50301)
0928 21:07:26.442133 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:26.442148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:26.442250 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:26.442516 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:26.442518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:26.442703 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:26.442799 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:26.442804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.443600 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.443603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:26.444204 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:26.444213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.444672 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.444675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:26.444738 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:26.445220 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:26.445231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:26.447314 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
W20260928 21:07:26.447703 31321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:26.441939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50301 (local address 127.21.140.126:37743)
0928 21:07:26.442121 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:26.442125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:26.442225 (+ 100us) server_negotiation.cc:407] Connection header received
0928 21:07:26.442343 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:26.442345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:26.442395 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:26.442477 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:26.442935 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.443468 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.444322 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:26.444562 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:26.444644 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:26.447547 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:50301: BlockingRecv error: recv got EOF from 127.21.140.67:50301 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 21:07:27 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:27 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:27.036345 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:27.030530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:60769)
0928 21:07:27.030889 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:27.030903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:27.031010 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:27.031243 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:27.031245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:27.031431 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:27.031531 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:27.031537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.032348 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.032351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:27.032929 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:27.032937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.033458 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.033461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:27.033542 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:27.034164 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:27.034178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:27.036165 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
W20260928 21:07:27.036433 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:27.030648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:60769 (local address 127.21.140.126:37743)
0928 21:07:27.030978 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:27.030982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:27.030997 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:07:27.031083 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:27.031085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:27.031127 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:27.031202 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:27.031660 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.032230 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.033067 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.033304 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.033399 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:27.036305 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:60769: BlockingRecv error: recv got EOF from 127.21.140.65:60769 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":81,"threads_started":1}
Sep 28 21:07:27 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:27 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:27.237497 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:27.231784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:44079)
0928 21:07:27.232126 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:27.232147 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:27.232276 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:27.232553 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:27.232556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:27.232772 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:27.232861 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:27.232866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.233638 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.233642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:27.234240 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:27.234248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.234700 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.234703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:27.234791 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:27.235304 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:27.235314 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:27.237318 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":94,"threads_started":1}
W20260928 21:07:27.237615 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:27.231921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:44079 (local address 127.21.140.126:37743)
0928 21:07:27.232088 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:27.232093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:27.232224 (+ 131us) server_negotiation.cc:407] Connection header received
0928 21:07:27.232393 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:27.232400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:27.232444 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:27.232533 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:27.232996 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.233520 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.234362 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.234585 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.234697 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:27.237482 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:44079: BlockingRecv error: recv got EOF from 127.21.140.66:44079 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 21:07:27 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:27 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.67@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:27.454480 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:27.448540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.67:50519)
0928 21:07:27.448798 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:27.448812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:27.448924 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:27.449192 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:27.449195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:27.449428 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:27.449526 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:27.449532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.450550 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.450554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:27.451120 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:27.451129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.451607 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.451610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:27.451699 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:27.452232 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:27.452244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:27.454281 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
W20260928 21:07:27.454555 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:27.448578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50519 (local address 127.21.140.126:37743)
0928 21:07:27.448739 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:27.448744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:27.448892 (+ 148us) server_negotiation.cc:407] Connection header received
0928 21:07:27.449009 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:27.449012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:27.449059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:27.449136 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:27.449641 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.450425 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.451250 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:27.451482 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:27.451593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:27.454433 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.67:50519: BlockingRecv error: recv got EOF from 127.21.140.67:50519 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 21:07:28 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:28 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.65@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:28.043159 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.037381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.65:55693)
0928 21:07:28.037674 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:28.037692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:28.037807 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:28.038113 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:28.038116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:28.038327 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:28.038418 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.038423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.039218 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.039222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.039782 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.039790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.040265 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.040268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.040361 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.040908 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:28.040920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:28.043025 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
W20260928 21:07:28.043386 31329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:28.037423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:55693 (local address 127.21.140.126:37743)
0928 21:07:28.037683 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:28.037687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:28.037765 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:07:28.037891 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:28.037893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:28.037934 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:28.038092 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:28.038550 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.039104 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.039901 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.040141 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.040250 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.043226 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.65:55693: BlockingRecv error: recv got EOF from 127.21.140.65:55693 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
Sep 28 21:07:28 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:07:28 dist-test-slave-d5wc krb5kdc[30269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.140.66@KRBTEST.COM for oryx/127.21.140.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:07:28.244129 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.238459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37743 (local address 127.21.140.66:59353)
0928 21:07:28.238804 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:28.238823 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:28.239013 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:28.239288 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:28.239291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:28.239501 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:28.239590 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.239596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.240369 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.240373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.240982 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.240991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.241461 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.241464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.241548 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.242099 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:28.242111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:28.243958 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.140.126:37743: Server oryx/127.21.140.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
W20260928 21:07:28.244189 31329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:28.238600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:59353 (local address 127.21.140.126:37743)
0928 21:07:28.238770 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:28.238774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:28.238952 (+ 178us) server_negotiation.cc:407] Connection header received
0928 21:07:28.239118 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:28.239122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:28.239172 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:28.239266 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:28.239720 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.240254 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.241105 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.241342 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.241459 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.244083 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.140.66:59353: BlockingRecv error: recv got EOF from 127.21.140.66:59353 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 21:07:28.318209 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 30854
I20260928 21:07:28.323761 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31006
I20260928 21:07:28.329622 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31138
I20260928 21:07:28.335188 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 30784
2026-09-28T21:07:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17997 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T21:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:28Z Disabled control of system clock
I20260928 21:07:28.365908 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:42963
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:07:28.476904 31334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:28.477190 31334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:28.477224 31334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:28.481473 31334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:28.481540 31334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:28.481556 31334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:28.481573 31334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:28.481590 31334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:28.486505 31334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:42963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:28.487635 31334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:28.489053 31334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:28.498100 31339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:28.498203 31334 server_base.cc:1034] running on GCE node
W20260928 21:07:28.498139 31342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:28.498152 31340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:28.498955 31334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:28.500098 31334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:28.501320 31334 hybrid_clock.cc:648] HybridClock initialized: now 1790629648501285 us; error 31 us; skew 500 ppm
I20260928 21:07:28.503271 31334 webserver.cc:460] Webserver started at http://127.21.140.126:43545/ using document root <none> and password file <none>
I20260928 21:07:28.503901 31334 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:28.503999 31334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:28.504238 31334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:28.506390 31334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "d6b0c8e7f49e499595eaf5a956d316d3"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "1ca0c38948a6305b5f639079a97b8e60"
server_key_iv: "e2448c22429e44a2cb8e83b9d2350ced"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.506987 31334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "d6b0c8e7f49e499595eaf5a956d316d3"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "1ca0c38948a6305b5f639079a97b8e60"
server_key_iv: "e2448c22429e44a2cb8e83b9d2350ced"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.510927 31334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:07:28.513538 31348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.514901 31334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:28.515056 31334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "d6b0c8e7f49e499595eaf5a956d316d3"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "1ca0c38948a6305b5f639079a97b8e60"
server_key_iv: "e2448c22429e44a2cb8e83b9d2350ced"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.515177 31334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:28.541469 31334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:28.542253 31334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:28.542465 31334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:28.549706 31334 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:42963
I20260928 21:07:28.549738 31400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:42963 every 8 connection(s)
I20260928 21:07:28.550763 31334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:07:28.552033 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31334
I20260928 21:07:28.552126 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:07:28.552474 22065 external_mini_cluster.cc:1428] Setting key 368ae9a3628c1a717549ba538351a44a
I20260928 21:07:28.554527 31401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:28.560613 31401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3: Bootstrap starting.
I20260928 21:07:28.562485 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.553923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.21.140.126:42963)
0928 21:07:28.554603 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:28.554620 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:28.554693 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:07:28.555354 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:28.555373 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:28.555749 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:28.556116 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:28.556575 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.557542 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.558835 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.559331 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.559483 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.560495 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:28.560500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:28.560513 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:28.561016 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:28.561076 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:28.561097 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:28.561184 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:28.561387 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 21:07:28.561678 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260928 21:07:28.563059 31401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:28.563848 31401 log.cc:824] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:28.565840 31401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3: No bootstrap required, opened a new log
I20260928 21:07:28.569279 31401 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42963 } }
I20260928 21:07:28.569561 31401 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:28.569638 31401 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b0c8e7f49e499595eaf5a956d316d3, State: Initialized, Role: FOLLOWER
I20260928 21:07:28.570139 31401 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42963 } }
I20260928 21:07:28.570297 31401 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:07:28.570384 31401 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:07:28.570509 31401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:28.571461 31401 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42963 } }
I20260928 21:07:28.571798 31401 leader_election.cc:302] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6b0c8e7f49e499595eaf5a956d316d3; no voters:
I20260928 21:07:28.572127 31401 leader_election.cc:288] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:07:28.572288 31406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:28.572536 31406 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [term 1 LEADER]: Becoming Leader. State: Replica: d6b0c8e7f49e499595eaf5a956d316d3, State: Running, Role: LEADER
I20260928 21:07:28.572875 31406 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42963 } }
I20260928 21:07:28.573347 31401 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:28.574657 31408 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b0c8e7f49e499595eaf5a956d316d3. Latest consensus state: current_term: 1 leader_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42963 } } }
I20260928 21:07:28.574680 31407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b0c8e7f49e499595eaf5a956d316d3" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42963 } } }
I20260928 21:07:28.574884 31408 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:28.574915 31407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3 [sys.catalog]: This master's current role is: LEADER
W20260928 21:07:28.578440 31421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:07:28.578503 31421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:07:28.578585 31416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:28.579214 31416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:28.584782 31416 catalog_manager.cc:1347] Generated new cluster ID: d1d8e4bdc9084855a6fb0c5ea35a935f
I20260928 21:07:28.584862 31416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:28.593057 31416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:07:28.593998 31416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:28.598924 31416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6b0c8e7f49e499595eaf5a956d316d3: Generated new TSK 0
I20260928 21:07:28.599704 31416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:07:28.611186 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:42963
--builtin_ntp_servers=127.21.140.84:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:07:28.721369 31425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:28.721634 31425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:28.721671 31425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:28.726097 31425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:28.726169 31425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:28.726267 31425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:28.731299 31425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:28.732436 31425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:28.733844 31425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:28.743611 31430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:28.743584 31433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:28.743799 31425 server_base.cc:1034] running on GCE node
W20260928 21:07:28.743597 31431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:28.744364 31425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:28.745012 31425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:28.746225 31425 hybrid_clock.cc:648] HybridClock initialized: now 1790629648746192 us; error 55 us; skew 500 ppm
I20260928 21:07:28.748093 31425 webserver.cc:460] Webserver started at http://127.21.140.65:35315/ using document root <none> and password file <none>
I20260928 21:07:28.748696 31425 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:28.748796 31425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:28.749032 31425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:28.751138 31425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "e2c7572a37914b8699c288fd94075a4b"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "24b6caf41a519c4cf2d9f1a60188663e"
server_key_iv: "893e7308c936a081527725a4ca22f14f"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.751714 31425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c7572a37914b8699c288fd94075a4b"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "24b6caf41a519c4cf2d9f1a60188663e"
server_key_iv: "893e7308c936a081527725a4ca22f14f"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.755349 31425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:07:28.757959 31439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.759321 31425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:28.759483 31425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "e2c7572a37914b8699c288fd94075a4b"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "24b6caf41a519c4cf2d9f1a60188663e"
server_key_iv: "893e7308c936a081527725a4ca22f14f"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.759604 31425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:28.773242 31425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:28.774133 31425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:28.774338 31425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:28.775014 31425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:28.775982 31425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:28.776057 31425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.776122 31425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:28.776152 31425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.784797 31425 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:42407
I20260928 21:07:28.784863 31552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:42407 every 8 connection(s)
I20260928 21:07:28.785732 31425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:28.787895 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31425
I20260928 21:07:28.788013 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:07:28.788344 22065 external_mini_cluster.cc:1428] Setting key 0e9ce0de307bb666d8f3db8c2ba24c14
I20260928 21:07:28.790524 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:42963
--builtin_ntp_servers=127.21.140.84:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:07:28.796959 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.787571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:41059 (local address 127.21.140.126:42963)
0928 21:07:28.787749 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:28.787754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:28.788927 (+ 1173us) server_negotiation.cc:407] Connection header received
0928 21:07:28.790745 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:28.790750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:28.790815 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:28.790948 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:28.792160 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.793053 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.794587 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.795043 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.795175 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.796319 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:28.796326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:28.796333 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:28.796352 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:28.796378 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:28.796390 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:28.796480 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:28.796623 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:07:28.796730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 21:07:28.797667 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.787977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:42963 (local address 127.21.140.65:41059)
0928 21:07:28.788780 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:28.788822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:28.790487 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:28.791157 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:28.791169 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:28.791277 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:28.791996 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.792018 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.793213 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.793216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.794426 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.794437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.795212 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.795215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.795375 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.796050 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:07:28.796052 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:28.796091 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:07:28.796153 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:28.796482 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:28.796488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:28.796492 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:07:28.796587 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:07:28.796805 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":111,"threads_started":1}
I20260928 21:07:28.798940 31553 heartbeater.cc:347] Connected to a master server at 127.21.140.126:42963
I20260928 21:07:28.799160 31553 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:28.799666 31553 heartbeater.cc:511] Master 127.21.140.126:42963 requested a full tablet report, sending...
I20260928 21:07:28.801081 31365 ts_manager.cc:194] Registered new tserver with Master: e2c7572a37914b8699c288fd94075a4b (127.21.140.65:42407)
I20260928 21:07:28.802278 31365 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.65:41059
W20260928 21:07:28.904717 31556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:28.904974 31556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:28.905010 31556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:28.909241 31556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:28.909299 31556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:28.909353 31556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:28.914307 31556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:28.915381 31556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:28.916800 31556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:28.926481 31562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:28.926573 31556 server_base.cc:1034] running on GCE node
W20260928 21:07:28.926716 31564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:28.926731 31561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:28.927203 31556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:28.927863 31556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:28.929072 31556 hybrid_clock.cc:648] HybridClock initialized: now 1790629648929058 us; error 29 us; skew 500 ppm
I20260928 21:07:28.930977 31556 webserver.cc:460] Webserver started at http://127.21.140.66:38081/ using document root <none> and password file <none>
I20260928 21:07:28.931560 31556 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:28.931656 31556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:28.931890 31556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:28.933902 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "0291e81eedba45acb24803734644683b"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "a73b860be660f90dd8cb190090b14a47"
server_key_iv: "7eb318e43761bfe8df5ef4dfe9c74bd2"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.934522 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "0291e81eedba45acb24803734644683b"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "a73b860be660f90dd8cb190090b14a47"
server_key_iv: "7eb318e43761bfe8df5ef4dfe9c74bd2"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.938294 31556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:07:28.940907 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.942436 31556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:28.942582 31556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "0291e81eedba45acb24803734644683b"
format_stamp: "Formatted at 2026-09-28 21:07:28 on dist-test-slave-d5wc"
server_key: "a73b860be660f90dd8cb190090b14a47"
server_key_iv: "7eb318e43761bfe8df5ef4dfe9c74bd2"
server_key_version: "encryptionkey@0"
I20260928 21:07:28.942705 31556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:28.955554 31556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:28.956248 31556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:28.956460 31556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:28.957087 31556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:28.958102 31556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:28.958177 31556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.958251 31556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:28.958295 31556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:28.967736 31556 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:40081
I20260928 21:07:28.967757 31683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:40081 every 8 connection(s)
I20260928 21:07:28.968657 31556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:28.977530 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31556
I20260928 21:07:28.977666 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:07:28.978120 22065 external_mini_cluster.cc:1428] Setting key 8d11ac21cc4ad327f2e1332aba9b606d
I20260928 21:07:28.978161 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.970724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:49285 (local address 127.21.140.126:42963)
0928 21:07:28.970929 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:28.970933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:28.971876 (+ 943us) server_negotiation.cc:407] Connection header received
0928 21:07:28.973379 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:28.973383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:28.973435 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:28.973514 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:28.974554 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.975114 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.976263 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.976482 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.976565 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.977581 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:28.977584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:28.977586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:28.977598 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:28.977616 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:28.977623 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:28.977726 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:28.977914 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:07:28.978039 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 21:07:28.979154 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:28.971032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:42963 (local address 127.21.140.66:49285)
0928 21:07:28.971705 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:28.971748 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:28.973063 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:28.973673 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:28.973680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:28.973764 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:28.974417 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.974437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.975226 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.975229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.976158 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:28.976167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:28.976616 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:28.976618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:28.976752 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:28.977308 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:07:28.977310 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:28.977347 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:07:28.977407 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:28.977740 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:28.977751 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:28.977758 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:07:28.977907 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 21:07:28.978235 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
I20260928 21:07:28.980221 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:42963
--builtin_ntp_servers=127.21.140.84:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:07:28.980333 31684 heartbeater.cc:347] Connected to a master server at 127.21.140.126:42963
I20260928 21:07:28.980523 31684 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:28.980978 31684 heartbeater.cc:511] Master 127.21.140.126:42963 requested a full tablet report, sending...
I20260928 21:07:28.982211 31365 ts_manager.cc:194] Registered new tserver with Master: 0291e81eedba45acb24803734644683b (127.21.140.66:40081)
I20260928 21:07:28.982787 31365 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.66:49285
W20260928 21:07:29.093598 31687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:29.093909 31687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:29.093997 31687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:29.098091 31687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:29.098165 31687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:29.098254 31687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:29.102990 31687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:29.104081 31687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:29.105463 31687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:29.115528 31695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.115525 31687 server_base.cc:1034] running on GCE node
W20260928 21:07:29.115581 31692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.115521 31693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.116199 31687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:29.116779 31687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:29.117966 31687 hybrid_clock.cc:648] HybridClock initialized: now 1790629649117931 us; error 44 us; skew 500 ppm
I20260928 21:07:29.120008 31687 webserver.cc:460] Webserver started at http://127.21.140.67:41605/ using document root <none> and password file <none>
I20260928 21:07:29.120609 31687 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:29.120671 31687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:29.120922 31687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:29.123045 31687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "87b700d4f61b4fb197e4048aad8cfedc"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "646a32b983d435d87a9d7ca52edceac3"
server_key_iv: "841d21346437fac47d89a7a1d5d4a678"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.123600 31687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "87b700d4f61b4fb197e4048aad8cfedc"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "646a32b983d435d87a9d7ca52edceac3"
server_key_iv: "841d21346437fac47d89a7a1d5d4a678"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.127264 31687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:07:29.129825 31701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.131045 31687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:07:29.131170 31687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "87b700d4f61b4fb197e4048aad8cfedc"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "646a32b983d435d87a9d7ca52edceac3"
server_key_iv: "841d21346437fac47d89a7a1d5d4a678"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.131301 31687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:29.155198 31687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:29.155894 31687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:29.156095 31687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:29.156672 31687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:29.157701 31687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:29.157775 31687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.157847 31687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:29.157893 31687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.166791 31687 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:40551
I20260928 21:07:29.166818 31814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:40551 every 8 connection(s)
I20260928 21:07:29.167696 31687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:07:29.177021 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31687
I20260928 21:07:29.177140 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:07:29.177470 22065 external_mini_cluster.cc:1428] Setting key 4e401893a9fe1ff250b7568f04f6c0e9
I20260928 21:07:29.177606 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.169557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:34967 (local address 127.21.140.126:42963)
0928 21:07:29.169722 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:29.169727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:29.170943 (+ 1216us) server_negotiation.cc:407] Connection header received
0928 21:07:29.172306 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:29.172309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:29.172360 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:29.172459 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:29.173543 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:29.174251 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:29.175637 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:29.175860 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:29.175955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:29.177026 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:29.177031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:29.177035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:29.177053 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:29.177079 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:29.177093 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:29.177180 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:29.177373 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:07:29.177442 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:07:29.178743 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.170059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:42963 (local address 127.21.140.67:34967)
0928 21:07:29.170735 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:29.170778 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:29.172066 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:29.172633 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:29.172640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:29.172726 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:29.173343 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:29.173378 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:29.174399 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:29.174402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:29.175502 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:29.175513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:29.176006 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:29.176008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:29.176149 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:29.176717 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:07:29.176718 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:29.176759 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:07:29.176825 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:29.177194 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:29.177204 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:29.177210 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:07:29.177333 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:07:29.177628 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260928 21:07:29.180006 31815 heartbeater.cc:347] Connected to a master server at 127.21.140.126:42963
I20260928 21:07:29.180253 31815 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:29.180815 31815 heartbeater.cc:511] Master 127.21.140.126:42963 requested a full tablet report, sending...
I20260928 21:07:29.181846 31365 ts_manager.cc:194] Registered new tserver with Master: 87b700d4f61b4fb197e4048aad8cfedc (127.21.140.67:40551)
I20260928 21:07:29.182551 31365 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.67:34967
I20260928 21:07:29.191659 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:07:29.194556 31404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:29.193604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.21.140.126:42963)
0928 21:07:29.193774 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:29.193780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:29.193901 (+ 121us) server_negotiation.cc:407] Connection header received
0928 21:07:29.194001 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:29.194007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:29.194062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:29.194123 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:29.194429 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35538: BlockingRecv error: recv got EOF from 127.0.0.1:35538 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 21:07:29.194654 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31425
I20260928 21:07:29.200378 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31556
I20260928 21:07:29.206195 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31687
I20260928 21:07:29.211412 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31334
2026-09-28T21:07:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (875 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T21:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:29Z Disabled control of system clock
I20260928 21:07:29.241107 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44689
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:07:29.354414 31826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:29.354710 31826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:29.354748 31826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:29.358783 31826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:29.358846 31826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:29.358862 31826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:29.358880 31826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:29.358896 31826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:29.363727 31826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:29.364804 31826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:29.366214 31826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:29.375113 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.375133 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.375113 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.375607 31826 server_base.cc:1034] running on GCE node
I20260928 21:07:29.376479 31826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:29.378283 31826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:29.379508 31826 hybrid_clock.cc:648] HybridClock initialized: now 1790629649379491 us; error 40 us; skew 500 ppm
I20260928 21:07:29.382258 31826 webserver.cc:460] Webserver started at http://127.21.140.126:36259/ using document root <none> and password file <none>
I20260928 21:07:29.383006 31826 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:29.383087 31826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:29.383266 31826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:29.385327 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "95530732ff26485cbe84c11555d71305"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "c1bdf07ca17f76c9ac88a16ebecfa0e5"
server_key_iv: "e43da7d79a89c19bf4fe3ccd4f9cfc90"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.385851 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "95530732ff26485cbe84c11555d71305"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "c1bdf07ca17f76c9ac88a16ebecfa0e5"
server_key_iv: "e43da7d79a89c19bf4fe3ccd4f9cfc90"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.389616 31826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 21:07:29.392263 31840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.393569 31826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:29.393674 31826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "95530732ff26485cbe84c11555d71305"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "c1bdf07ca17f76c9ac88a16ebecfa0e5"
server_key_iv: "e43da7d79a89c19bf4fe3ccd4f9cfc90"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.393839 31826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:29.398541 31826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:29.399165 31826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:29.399351 31826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:29.406279 31826 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44689
I20260928 21:07:29.406316 31892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44689 every 8 connection(s)
I20260928 21:07:29.407284 31826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:07:29.410614 31893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:29.416442 31893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305: Bootstrap starting.
I20260928 21:07:29.416828 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31826
I20260928 21:07:29.416958 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:07:29.417284 22065 external_mini_cluster.cc:1428] Setting key eb97da568b555ce386a28b4494e58acf
I20260928 21:07:29.419400 31893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:29.420166 31893 log.cc:824] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:29.422648 31893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305: No bootstrap required, opened a new log
I20260928 21:07:29.423470 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.418895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.21.140.126:44689)
0928 21:07:29.419560 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:29.419577 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:29.419612 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:07:29.420276 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:29.420308 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:29.420752 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:29.421140 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:29.422060 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:29.422065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:29.422078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:29.422089 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:29.422144 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:29.422161 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:29.422252 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:29.422400 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 21:07:29.422682 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260928 21:07:29.426429 31893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95530732ff26485cbe84c11555d71305" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44689 } }
I20260928 21:07:29.426666 31893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:29.426699 31893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95530732ff26485cbe84c11555d71305, State: Initialized, Role: FOLLOWER
I20260928 21:07:29.427145 31893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95530732ff26485cbe84c11555d71305" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44689 } }
I20260928 21:07:29.427266 31893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:07:29.427356 31893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:07:29.427488 31893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:29.428449 31893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95530732ff26485cbe84c11555d71305" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44689 } }
I20260928 21:07:29.428884 31893 leader_election.cc:302] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95530732ff26485cbe84c11555d71305; no voters:
I20260928 21:07:29.429184 31893 leader_election.cc:288] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:07:29.429354 31898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:29.429620 31898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [term 1 LEADER]: Becoming Leader. State: Replica: 95530732ff26485cbe84c11555d71305, State: Running, Role: LEADER
I20260928 21:07:29.430056 31898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95530732ff26485cbe84c11555d71305" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44689 } }
I20260928 21:07:29.430574 31893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:29.431772 31900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95530732ff26485cbe84c11555d71305. Latest consensus state: current_term: 1 leader_uuid: "95530732ff26485cbe84c11555d71305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95530732ff26485cbe84c11555d71305" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44689 } } }
I20260928 21:07:29.431772 31899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95530732ff26485cbe84c11555d71305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95530732ff26485cbe84c11555d71305" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44689 } } }
I20260928 21:07:29.431934 31900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:29.431946 31899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305 [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:29.432461 31907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:29.436113 31907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:29.441495 31907 catalog_manager.cc:1347] Generated new cluster ID: 86f9aa1595bd4942b601ab05b146f56a
I20260928 21:07:29.441571 31907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:29.458010 31907 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:07:29.458424 31907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:29.464453 31907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95530732ff26485cbe84c11555d71305: Generated new TSK 0
I20260928 21:07:29.465282 31907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:07:29.479902 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44689
--builtin_ntp_servers=127.21.140.84:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:07:29.593422 31917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:29.593713 31917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:29.593755 31917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:29.598275 31917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:29.598340 31917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:29.598431 31917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:29.603464 31917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:29.604581 31917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:29.605997 31917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:29.615888 31923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.615984 31922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.615883 31925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.616106 31917 server_base.cc:1034] running on GCE node
I20260928 21:07:29.616626 31917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:29.617357 31917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:29.618530 31917 hybrid_clock.cc:648] HybridClock initialized: now 1790629649618502 us; error 44 us; skew 500 ppm
I20260928 21:07:29.620386 31917 webserver.cc:460] Webserver started at http://127.21.140.65:37873/ using document root <none> and password file <none>
I20260928 21:07:29.621060 31917 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:29.621151 31917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:29.621328 31917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:29.623641 31917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "e39abf49f9b54507af205f38f0a9f15f"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "af5783dd06283c7a5740c952e3874fb8"
server_key_iv: "8410affa2fda2d7a5e65f9a01c0285b5"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.624229 31917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "e39abf49f9b54507af205f38f0a9f15f"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "af5783dd06283c7a5740c952e3874fb8"
server_key_iv: "8410affa2fda2d7a5e65f9a01c0285b5"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.628114 31917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:07:29.630851 31931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.632524 31917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:07:29.632630 31917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "e39abf49f9b54507af205f38f0a9f15f"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "af5783dd06283c7a5740c952e3874fb8"
server_key_iv: "8410affa2fda2d7a5e65f9a01c0285b5"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.632786 31917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:29.638043 31917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:29.638630 31917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:29.638779 31917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:29.639295 31917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:29.640218 31917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:29.640265 31917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.640308 31917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:29.640324 31917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.649776 31917 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:38699
I20260928 21:07:29.649796 32044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:38699 every 8 connection(s)
I20260928 21:07:29.650728 31917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:29.656831 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.652574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:35319 (local address 127.21.140.126:44689)
0928 21:07:29.652709 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:29.652713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:29.653787 (+ 1074us) server_negotiation.cc:407] Connection header received
0928 21:07:29.655092 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:29.655094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:29.655133 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:29.655212 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:29.656267 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:29.656270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:29.656272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:29.656285 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:29.656314 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:29.656322 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:29.656403 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:29.656584 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:07:29.656658 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:07:29.656936 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 31917
I20260928 21:07:29.657034 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:07:29.657327 22065 external_mini_cluster.cc:1428] Setting key 857da9f72c0216507d6ae378c9ad6592
I20260928 21:07:29.657629 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.652972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44689 (local address 127.21.140.65:35319)
0928 21:07:29.653635 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:29.653676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:29.654899 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:29.655367 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:29.655373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:29.655478 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:29.656003 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:07:29.656004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:29.656044 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:07:29.656110 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:29.656415 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:29.656421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:29.656425 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:07:29.656576 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 21:07:29.656760 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260928 21:07:29.658857 32045 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44689
I20260928 21:07:29.659128 32045 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:29.659415 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44689
--builtin_ntp_servers=127.21.140.84:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:07:29.659548 32045 heartbeater.cc:511] Master 127.21.140.126:44689 requested a full tablet report, sending...
I20260928 21:07:29.661156 31857 ts_manager.cc:194] Registered new tserver with Master: e39abf49f9b54507af205f38f0a9f15f (127.21.140.65:38699)
W20260928 21:07:29.776309 32048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:29.776593 32048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:29.776635 32048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:29.780886 32048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:29.780956 32048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:29.781015 32048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:29.786196 32048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:29.787350 32048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:29.788746 32048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:29.799232 32054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.799268 32056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.799268 32053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.799371 32048 server_base.cc:1034] running on GCE node
I20260928 21:07:29.799891 32048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:29.800586 32048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:29.801810 32048 hybrid_clock.cc:648] HybridClock initialized: now 1790629649801792 us; error 32 us; skew 500 ppm
I20260928 21:07:29.803967 32048 webserver.cc:460] Webserver started at http://127.21.140.66:45319/ using document root <none> and password file <none>
I20260928 21:07:29.804605 32048 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:29.804705 32048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:29.804973 32048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:29.807242 32048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "d3fb3ee1fad44804a84a1853ee9b1291"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "ce9c97a05982080dff1b47aff063cf6d"
server_key_iv: "1490293cdb3a484aa2a061770afe4a09"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.807871 32048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "d3fb3ee1fad44804a84a1853ee9b1291"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "ce9c97a05982080dff1b47aff063cf6d"
server_key_iv: "1490293cdb3a484aa2a061770afe4a09"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.811931 32048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:07:29.814729 32062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.816248 32048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:07:29.816401 32048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "d3fb3ee1fad44804a84a1853ee9b1291"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "ce9c97a05982080dff1b47aff063cf6d"
server_key_iv: "1490293cdb3a484aa2a061770afe4a09"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.816545 32048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:29.821410 32048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:29.822101 32048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:29.822288 32048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:29.822836 32048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:29.823752 32048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:29.823823 32048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.823904 32048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:29.823957 32048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.833308 32048 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:44581
I20260928 21:07:29.833366 32175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:44581 every 8 connection(s)
I20260928 21:07:29.834472 32048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:29.836742 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 32048
I20260928 21:07:29.836872 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:07:29.837211 22065 external_mini_cluster.cc:1428] Setting key e4b6bd8a73a82227d5316d85da49e547
I20260928 21:07:29.839635 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44689
--builtin_ntp_servers=127.21.140.84:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:07:29.842051 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.836468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:32953 (local address 127.21.140.126:44689)
0928 21:07:29.836611 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:29.836623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:29.838296 (+ 1673us) server_negotiation.cc:407] Connection header received
0928 21:07:29.839913 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:29.839918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:29.839968 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:29.840042 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:29.841527 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:29.841531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:29.841533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:29.841546 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:29.841568 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:29.841582 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:29.841674 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:29.841838 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 21:07:29.841884 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:07:29.842875 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:29.837004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44689 (local address 127.21.140.66:32953)
0928 21:07:29.838152 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:29.838199 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:29.839589 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:29.840282 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:29.840294 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:29.840421 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:29.841192 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:07:29.841196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:29.841257 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:07:29.841366 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:29.841670 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:29.841679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:29.841688 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:07:29.841866 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 21:07:29.842131 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":140,"threads_started":1}
I20260928 21:07:29.843988 32176 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44689
I20260928 21:07:29.844233 32176 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:29.844715 32176 heartbeater.cc:511] Master 127.21.140.126:44689 requested a full tablet report, sending...
I20260928 21:07:29.846071 31857 ts_manager.cc:194] Registered new tserver with Master: d3fb3ee1fad44804a84a1853ee9b1291 (127.21.140.66:44581)
W20260928 21:07:29.954895 32179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:29.955206 32179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:29.955286 32179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:29.959551 32179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:29.959636 32179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:29.959728 32179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:29.964805 32179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:29.965874 32179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:29.967327 32179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:29.976902 32185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.977101 32179 server_base.cc:1034] running on GCE node
W20260928 21:07:29.976955 32187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:29.977005 32184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:29.977670 32179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:29.978392 32179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:29.979598 32179 hybrid_clock.cc:648] HybridClock initialized: now 1790629649979579 us; error 31 us; skew 500 ppm
I20260928 21:07:29.981628 32179 webserver.cc:460] Webserver started at http://127.21.140.67:37487/ using document root <none> and password file <none>
I20260928 21:07:29.982331 32179 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:29.982426 32179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:29.982666 32179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:29.984851 32179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "02a9afde8ee94b64a4a5547a6d213fb6"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "bf7e2a3b456c4235a89f7b35faa3cc6a"
server_key_iv: "464efd3f92b45eaae649947b0c9e9523"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.985438 32179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "02a9afde8ee94b64a4a5547a6d213fb6"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "bf7e2a3b456c4235a89f7b35faa3cc6a"
server_key_iv: "464efd3f92b45eaae649947b0c9e9523"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.989339 32179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260928 21:07:29.992038 32193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:29.993386 32179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:29.993541 32179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "02a9afde8ee94b64a4a5547a6d213fb6"
format_stamp: "Formatted at 2026-09-28 21:07:29 on dist-test-slave-d5wc"
server_key: "bf7e2a3b456c4235a89f7b35faa3cc6a"
server_key_iv: "464efd3f92b45eaae649947b0c9e9523"
server_key_version: "encryptionkey@0"
I20260928 21:07:29.993671 32179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:29.998667 32179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:29.999311 32179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:29.999516 32179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:30.000084 32179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:30.001057 32179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:30.001132 32179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:30.001202 32179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:30.001247 32179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:30.010182 32179 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:41629
I20260928 21:07:30.010243 32306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:41629 every 8 connection(s)
I20260928 21:07:30.011228 32179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:07:30.018205 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 32179
I20260928 21:07:30.018365 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:07:30.018473 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:30.013250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:37169 (local address 127.21.140.126:44689)
0928 21:07:30.013451 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:30.013455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:30.014619 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 21:07:30.016473 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:30.016476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:30.016534 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:30.016651 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:30.017896 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:30.017905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:30.017908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:07:30.017921 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:30.017948 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:07:30.017957 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:30.018111 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:30.018260 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:07:30.018309 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:07:30.018728 22065 external_mini_cluster.cc:1428] Setting key 955400116f46681f82b5511fd089e640
I20260928 21:07:30.019579 32309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:30.013670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44689 (local address 127.21.140.67:37169)
0928 21:07:30.014458 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:30.014501 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:30.015871 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:30.016848 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:30.016855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:30.016952 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:30.017580 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:07:30.017581 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:30.017629 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:07:30.017701 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:30.018099 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:30.018108 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:30.018114 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:07:30.018308 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 21:07:30.018566 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":120,"threads_started":1}
I20260928 21:07:30.020879 32307 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44689
I20260928 21:07:30.021143 32307 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:30.021579 32307 heartbeater.cc:511] Master 127.21.140.126:44689 requested a full tablet report, sending...
I20260928 21:07:30.022738 31857 ts_manager.cc:194] Registered new tserver with Master: 02a9afde8ee94b64a4a5547a6d213fb6 (127.21.140.67:41629)
I20260928 21:07:30.033120 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:07:30.036108 31896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:07:30.035177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.21.140.126:44689)
0928 21:07:30.035378 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:30.035382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:30.035422 (+ 40us) server_negotiation.cc:407] Connection header received
0928 21:07:30.035554 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:30.035556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:30.035590 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:30.035659 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:30.035963 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57406: BlockingRecv error: recv got EOF from 127.0.0.1:57406 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 21:07:30.036465 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31917
I20260928 21:07:30.042640 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 32048
I20260928 21:07:30.048391 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 32179
I20260928 21:07:30.053601 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 31826
2026-09-28T21:07:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (842 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:07:30 dist-test-slave-d5wc krb5kdc[32317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:07:30 dist-test-slave-d5wc krb5kdc[32317](info): set up 2 sockets
Sep 28 21:07:30 dist-test-slave-d5wc krb5kdc[32317](info): commencing operation
krb5kdc: starting...
W20260928 21:07:31.128309 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:07:31.253401 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:36507
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:31.361979 32330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:31.362268 32330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:31.362344 32330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:31.366329 32330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:07:31.366426 32330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:31.366463 32330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:31.366503 32330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:07:31.366539 32330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:07:31.371779 32330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:36507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:31.372867 32330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:31.373889 32330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:07:31.382733 32338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:31.382898 32330 server_base.cc:1034] running on GCE node
W20260928 21:07:31.382761 32335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:31.382750 32336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:31.384130 32330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:31.385614 32330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:31.386821 32330 hybrid_clock.cc:648] HybridClock initialized: now 1790629651386811 us; error 34 us; skew 500 ppm
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:31.389722 32330 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:07:31.390741 32330 webserver.cc:460] Webserver started at http://127.21.140.126:39189/ using document root <none> and password file <none>
I20260928 21:07:31.391310 32330 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:31.391372 32330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:31.391598 32330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:31.393651 32330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "52e2bcdbf97748b689ffbc71404b3c7b"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "edd81949be7b9ab7d91609f81002cd3e"
server_key_iv: "04e31e020c0d7a9a30005d1ee1eeb8b9"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.394237 32330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "52e2bcdbf97748b689ffbc71404b3c7b"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "edd81949be7b9ab7d91609f81002cd3e"
server_key_iv: "04e31e020c0d7a9a30005d1ee1eeb8b9"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.397943 32330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:07:31.400434 32345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.401767 32330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:31.401970 32330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "52e2bcdbf97748b689ffbc71404b3c7b"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "edd81949be7b9ab7d91609f81002cd3e"
server_key_iv: "04e31e020c0d7a9a30005d1ee1eeb8b9"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.402107 32330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:31.423457 32330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:31.426759 32330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:31.426982 32330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:31.434190 32330 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:36507
I20260928 21:07:31.434479 32397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:36507 every 8 connection(s)
I20260928 21:07:31.435231 32330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:07:31.437913 32398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:31.442186 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 32330
I20260928 21:07:31.442317 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:07:31.442571 22065 external_mini_cluster.cc:1428] Setting key c7f233639451b09df33c23d23a28e714
I20260928 21:07:31.443813 32398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b: Bootstrap starting.
I20260928 21:07:31.446524 32398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:31.447381 32398 log.cc:824] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:31.449493 32398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b: No bootstrap required, opened a new log
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:07:31.453478 32398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36507 } }
I20260928 21:07:31.453778 32398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:31.453867 32398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e2bcdbf97748b689ffbc71404b3c7b, State: Initialized, Role: FOLLOWER
I20260928 21:07:31.454362 32398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36507 } }
I20260928 21:07:31.454497 32398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:07:31.454602 32398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:07:31.454702 32398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:31.455581 32398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36507 } }
I20260928 21:07:31.455924 32398 leader_election.cc:302] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e2bcdbf97748b689ffbc71404b3c7b; no voters:
I20260928 21:07:31.456229 32398 leader_election.cc:288] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:07:31.456346 32403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:31.456575 32403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [term 1 LEADER]: Becoming Leader. State: Replica: 52e2bcdbf97748b689ffbc71404b3c7b, State: Running, Role: LEADER
I20260928 21:07:31.456923 32403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36507 } }
I20260928 21:07:31.457543 32398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:07:31.458596 32404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36507 } } }
I20260928 21:07:31.458721 32404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:31.458806 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:31.444086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.21.140.126:36507)
0928 21:07:31.444541 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:31.444560 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:31.444601 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:07:31.445482 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:31.445505 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:31.445918 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:31.446312 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:31.447091 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.448394 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.449297 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.449984 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.450142 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:31.452808 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:31.452814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:31.452832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:31.452874 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:31.455053 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:31.455515 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:31.455521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:31.455526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:31.455670 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:31.455963 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:31.455966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:31.455968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:31.456521 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:31.457298 (+ 777us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:31.457448 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:07:31.457739 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":164,"threads_started":1}
I20260928 21:07:31.458950 32404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e2bcdbf97748b689ffbc71404b3c7b. Latest consensus state: current_term: 1 leader_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e2bcdbf97748b689ffbc71404b3c7b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36507 } } }
I20260928 21:07:31.459033 32404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b [sys.catalog]: This master's current role is: LEADER
I20260928 21:07:31.459293 32412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:07:31.462399 32412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:07:31.468231 32412 catalog_manager.cc:1347] Generated new cluster ID: 8663b4b3773a402891db211756dab783
I20260928 21:07:31.468307 32412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:07:31.482153 32412 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:07:31.483106 32412 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:07:31.492748 32412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b: Generated new TSK 0
I20260928 21:07:31.493626 32412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:07:31.533063 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:36507
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:31.643705 32424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:31.643960 32424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:31.644002 32424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:31.648223 32424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:31.648276 32424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:31.648334 32424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:07:31.654145 32424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:31.655097 32424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:31.656060 32424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:07:31.665676 32424 server_base.cc:1034] running on GCE node
W20260928 21:07:31.665596 32429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:31.665623 32430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:31.665601 32432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:31.666381 32424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:31.666999 32424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:31.668212 32424 hybrid_clock.cc:648] HybridClock initialized: now 1790629651668202 us; error 33 us; skew 500 ppm
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:31.671031 32424 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:07:31.671958 32424 webserver.cc:460] Webserver started at http://127.21.140.65:33027/ using document root <none> and password file <none>
I20260928 21:07:31.672546 32424 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:31.672642 32424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:31.672957 32424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:31.675086 32424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "da6318aede454955800d8365da8bb70d"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "6ec40b9c90aa6a57e3387fbc937d0e89"
server_key_iv: "4a558723d488689a4e34cd02123e9cca"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.675668 32424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "da6318aede454955800d8365da8bb70d"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "6ec40b9c90aa6a57e3387fbc937d0e89"
server_key_iv: "4a558723d488689a4e34cd02123e9cca"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.679494 32424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:07:31.682035 32439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.683220 32424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:31.683372 32424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "da6318aede454955800d8365da8bb70d"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "6ec40b9c90aa6a57e3387fbc937d0e89"
server_key_iv: "4a558723d488689a4e34cd02123e9cca"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.683488 32424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:31.709841 32424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:31.712782 32424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:31.713048 32424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:31.713675 32424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:31.714569 32424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:31.714643 32424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.714713 32424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:31.714756 32424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.723582 32424 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:37729
I20260928 21:07:31.723608 32552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:37729 every 8 connection(s)
I20260928 21:07:31.724447 32424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:07:31.729609 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 32424
I20260928 21:07:31.729722 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:07:31.730099 22065 external_mini_cluster.cc:1428] Setting key 44ee21b6ba80407dc9125596b95724a3
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:07:31.741225 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:31.726364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:46251 (local address 127.21.140.126:36507)
0928 21:07:31.726498 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:31.726501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:31.727283 (+ 782us) server_negotiation.cc:407] Connection header received
0928 21:07:31.728586 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:31.728589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:31.728634 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:31.728710 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:31.730094 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.730799 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.734367 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.734781 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.734912 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:31.737513 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:31.737517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:31.737519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:31.737542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:31.739021 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:31.739546 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:31.739550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:31.739552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:31.739660 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:31.740018 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:31.740023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:31.740025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:31.740210 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:31.740317 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:31.740922 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 21:07:31.741051 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:07:31.742131 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:31.726662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:36507 (local address 127.21.140.65:46251)
0928 21:07:31.727125 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:31.727166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:31.728396 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:31.728878 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:31.728884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:31.729227 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:31.729921 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:31.729944 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.730979 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.730982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:31.734225 (+ 3243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:31.734236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.734954 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.734957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:31.735100 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:31.735765 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:31.735789 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:31.737323 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:31.739154 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:31.739160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:31.739175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:31.739439 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:31.739767 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:31.739771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:31.739773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:31.739919 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:31.740330 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:31.740345 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:31.740625 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 21:07:31.740867 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 21:07:31.741191 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260928 21:07:31.743592 32553 heartbeater.cc:347] Connected to a master server at 127.21.140.126:36507
I20260928 21:07:31.743891 32553 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:31.744482 32553 heartbeater.cc:511] Master 127.21.140.126:36507 requested a full tablet report, sending...
I20260928 21:07:31.746287 32362 ts_manager.cc:194] Registered new tserver with Master: da6318aede454955800d8365da8bb70d (127.21.140.65:37729)
I20260928 21:07:31.747465 32362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:46251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:07:31.759387 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:36507
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:31.869489 32558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:31.869750 32558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:31.869794 32558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:31.873808 32558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:31.873862 32558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:31.874074 32558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:07:31.879737 32558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:31.880739 32558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:31.881740 32558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:07:31.891471 32558 server_base.cc:1034] running on GCE node
W20260928 21:07:31.891453 32563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:31.891434 32564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:31.891438 32566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:31.892076 32558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:31.892645 32558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:31.893806 32558 hybrid_clock.cc:648] HybridClock initialized: now 1790629651893795 us; error 26 us; skew 500 ppm
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:31.896529 32558 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:07:31.897413 32558 webserver.cc:460] Webserver started at http://127.21.140.66:40891/ using document root <none> and password file <none>
I20260928 21:07:31.897953 32558 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:31.898087 32558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:31.898351 32558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:31.900277 32558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "efcd3660d07f440a9fa279503597865d"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "987b9dc06848813ae73db92afd791603"
server_key_iv: "c68280c947d8830d162b50181d1b36a4"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.900766 32558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "efcd3660d07f440a9fa279503597865d"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "987b9dc06848813ae73db92afd791603"
server_key_iv: "c68280c947d8830d162b50181d1b36a4"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.904346 32558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 21:07:31.906814 32573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.908262 32558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:07:31.908362 32558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "efcd3660d07f440a9fa279503597865d"
format_stamp: "Formatted at 2026-09-28 21:07:31 on dist-test-slave-d5wc"
server_key: "987b9dc06848813ae73db92afd791603"
server_key_iv: "c68280c947d8830d162b50181d1b36a4"
server_key_version: "encryptionkey@0"
I20260928 21:07:31.908509 32558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:31.928476 32558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:31.931870 32558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:31.932138 32558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:31.932742 32558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:31.933573 32558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:31.933620 32558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.933660 32558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:31.933715 32558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:31.942822 32558 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:40211
I20260928 21:07:31.942843 32686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:40211 every 8 connection(s)
I20260928 21:07:31.943707 32558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:07:31.945920 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 32558
I20260928 21:07:31.946070 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:07:31.946396 22065 external_mini_cluster.cc:1428] Setting key b251b7ea4262ab10cd179300d7533c29
Sep 28 21:07:31 dist-test-slave-d5wc krb5kdc[32317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629651, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:07:31.958319 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:31.945511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:54561 (local address 127.21.140.126:36507)
0928 21:07:31.945651 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:31.945655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:31.946634 (+ 979us) server_negotiation.cc:407] Connection header received
0928 21:07:31.948432 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:31.948435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:31.948485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:31.948567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:31.949716 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.950324 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.951491 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.951719 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.951834 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:31.954304 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:31.954308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:31.954310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:31.954333 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:31.955976 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:31.956523 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:31.956531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:31.956533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:31.956608 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:31.956963 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:31.956966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:31.956968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:31.957154 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:31.957283 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:31.957955 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 21:07:31.958144 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:07:31.959331 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:31.945887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:36507 (local address 127.21.140.66:54561)
0928 21:07:31.946424 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:31.946500 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:31.948177 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:31.948739 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:31.948746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:31.949096 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:31.949569 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:31.949582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.950450 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.950454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:31.951357 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:31.951367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:31.951870 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:31.951874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:31.952052 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:31.952656 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:31.952681 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:31.954133 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:31.956114 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:31.956120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:31.956135 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:31.956403 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:31.956702 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:31.956707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:31.956710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:31.956853 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:31.957272 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:31.957278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:31.957622 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 21:07:31.957863 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 21:07:31.958274 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":137,"threads_started":1}
I20260928 21:07:31.960747 32687 heartbeater.cc:347] Connected to a master server at 127.21.140.126:36507
I20260928 21:07:31.961040 32687 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:31.961660 32687 heartbeater.cc:511] Master 127.21.140.126:36507 requested a full tablet report, sending...
I20260928 21:07:31.962826 32362 ts_manager.cc:194] Registered new tserver with Master: efcd3660d07f440a9fa279503597865d (127.21.140.66:40211)
I20260928 21:07:31.963423 32362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:54561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:07:31.975042 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:36507
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:07:32.087218 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:07:32.087566 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:07:32.087651 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:07:32.091809 32692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:07:32.091867 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:07:32.091930 32692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:07:32.097267 32692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:07:32.098325 32692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:07:32.099308 32692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:07:32.109201 32692 server_base.cc:1034] running on GCE node
W20260928 21:07:32.109117 32698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:32.109117 32700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:07:32.109148 32697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:07:32.109776 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:07:32.110461 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:07:32.111687 32692 hybrid_clock.cc:648] HybridClock initialized: now 1790629652111646 us; error 47 us; skew 500 ppm
Sep 28 21:07:32 dist-test-slave-d5wc krb5kdc[32317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629652, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:07:32.114722 32692 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:07:32.115664 32692 webserver.cc:460] Webserver started at http://127.21.140.67:36151/ using document root <none> and password file <none>
I20260928 21:07:32.116238 32692 fs_manager.cc:359] Metadata directory not provided
I20260928 21:07:32.116333 32692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:07:32.116572 32692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:07:32.118700 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "72765ac170da47d685530f22b0017343"
format_stamp: "Formatted at 2026-09-28 21:07:32 on dist-test-slave-d5wc"
server_key: "0ffc3b15a94d629c4aa5967451423595"
server_key_iv: "e814a5eaff466af915bdb68e1bb05c47"
server_key_version: "encryptionkey@0"
I20260928 21:07:32.119261 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "72765ac170da47d685530f22b0017343"
format_stamp: "Formatted at 2026-09-28 21:07:32 on dist-test-slave-d5wc"
server_key: "0ffc3b15a94d629c4aa5967451423595"
server_key_iv: "e814a5eaff466af915bdb68e1bb05c47"
server_key_version: "encryptionkey@0"
I20260928 21:07:32.123090 32692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 21:07:32.125415 32707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:32.126639 32692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:07:32.126775 32692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "72765ac170da47d685530f22b0017343"
format_stamp: "Formatted at 2026-09-28 21:07:32 on dist-test-slave-d5wc"
server_key: "0ffc3b15a94d629c4aa5967451423595"
server_key_iv: "e814a5eaff466af915bdb68e1bb05c47"
server_key_version: "encryptionkey@0"
I20260928 21:07:32.126883 32692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:07:32.146214 32692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:07:32.149211 32692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:07:32.149421 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:07:32.150072 32692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:07:32.150876 32692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:07:32.150956 32692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:32.151057 32692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:07:32.151109 32692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:07:32.160365 32692 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:33769
I20260928 21:07:32.160386 352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:33769 every 8 connection(s)
I20260928 21:07:32.161271 32692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:07:32.161659 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 32692
I20260928 21:07:32.161785 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:07:32.162134 22065 external_mini_cluster.cc:1428] Setting key 25d6113f836748b6608fbc5e7b681fbf
Sep 28 21:07:32 dist-test-slave-d5wc krb5kdc[32317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629652, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:07:32.176129 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.163265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:52035 (local address 127.21.140.126:36507)
0928 21:07:32.163471 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.163476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.164544 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 21:07:32.165865 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.165867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.165917 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.166077 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:32.167195 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.167734 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.168940 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.169158 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.169245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.172004 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:32.172007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:32.172009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:32.172033 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:32.173671 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:32.174342 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:32.174345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:32.174347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:32.174460 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:32.174904 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:32.174907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:32.174908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:32.175113 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:32.175232 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.175780 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.175916 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:07:32.176936 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.163827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:36507 (local address 127.21.140.67:52035)
0928 21:07:32.164378 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.164434 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:32.165649 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:32.166215 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:32.166222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:32.166567 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:07:32.167039 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.167051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.167881 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.167884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.168804 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.168814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.169314 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.169318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.169465 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.170275 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:07:32.170292 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:07:32.171796 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:07:32.173807 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:32.173813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:32.173826 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:32.174217 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:32.174592 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:07:32.174595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:07:32.174596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:07:32.174765 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:07:32.175244 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:07:32.175252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:07:32.175535 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 21:07:32.175739 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 21:07:32.176090 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260928 21:07:32.178025 353 heartbeater.cc:347] Connected to a master server at 127.21.140.126:36507
I20260928 21:07:32.178278 353 heartbeater.cc:464] Registering TS with master...
I20260928 21:07:32.178764 353 heartbeater.cc:511] Master 127.21.140.126:36507 requested a full tablet report, sending...
I20260928 21:07:32.179790 32362 ts_manager.cc:194] Registered new tserver with Master: 72765ac170da47d685530f22b0017343 (127.21.140.67:33769)
I20260928 21:07:32.180395 32362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:52035
I20260928 21:07:32.188315 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:07:32.198137 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.190428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.21.140.126:36507)
0928 21:07:32.190573 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.190577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.190769 (+ 192us) server_negotiation.cc:407] Connection header received
0928 21:07:32.190959 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.190961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.191000 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.191088 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:07:32.191820 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.192346 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.193193 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.193416 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.193528 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.194989 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:07:32.194993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:07:32.194996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:07:32.195028 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:07:32.196324 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:32.196776 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:32.196780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:32.196782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:32.196876 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:07:32.197163 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:07:32.197168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:07:32.197169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:07:32.197412 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:07:32.197557 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.197806 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.197931 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:07:32.201355 32362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:07:32.204470 32362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:07:32.219508 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.213373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.21.140.67:33769)
0928 21:07:32.214939 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.214951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.214973 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:07:32.215045 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.215053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.215222 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.215408 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:32.215675 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.216401 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.217921 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.218764 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.218876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.219028 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.219215 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.219338 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":159,"threads_started":1}
I20260928 21:07:32.219475 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.213324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:37729 (local address 127.0.0.1:38640)
0928 21:07:32.214400 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.214413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:32.214522 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:32.215126 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:32.215132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:32.215157 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:32.215319 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.215326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.216840 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.216857 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.218048 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.218058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.219150 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.219157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.219239 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.219258 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:07:32.219321 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:07:32.219375 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260928 21:07:32.219475 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.213260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:33769 (local address 127.0.0.1:41798)
0928 21:07:32.213923 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.213999 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:32.214175 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:32.215389 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:32.215392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:32.215409 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:32.215553 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.215568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.216530 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.216535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.217802 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.217811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.218901 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.218907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.219011 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.219048 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:07:32.219167 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:07:32.219290 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":35,"thread_start_us":100,"threads_started":1}
I20260928 21:07:32.220003 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.214371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.21.140.66:40211)
0928 21:07:32.214823 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.214835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.214858 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:07:32.214935 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.214940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.215126 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.215270 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:32.215709 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.216791 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.218400 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.219232 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.219333 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.219456 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.219767 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.219862 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":150,"threads_started":1}
I20260928 21:07:32.219949 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.214198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:40211 (local address 127.0.0.1:35844)
0928 21:07:32.214482 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.214501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:32.214600 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:32.215441 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:32.215443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:32.215460 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:32.215580 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.215587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.216921 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.216925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.218263 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.218273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.219527 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.219533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.219624 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.219647 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:07:32.219724 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:07:32.219798 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":3,"thread_start_us":138,"threads_started":1}
I20260928 21:07:32.220220 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.213762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.21.140.65:37729)
0928 21:07:32.214168 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.214222 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.214623 (+ 401us) server_negotiation.cc:407] Connection header received
0928 21:07:32.214707 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.214712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.214959 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.215115 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:32.215459 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.216668 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.218183 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.218799 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.218939 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.219086 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.219917 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.220034 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":85,"threads_started":1}
I20260928 21:07:32.223155 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa48f4826ba44e99169dce15e0a0760 (DEFAULT_TABLE table=test-table [id=b9b51ebce45c4e55b2029079f6f927f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:32.223222 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa48f4826ba44e99169dce15e0a0760 (DEFAULT_TABLE table=test-table [id=b9b51ebce45c4e55b2029079f6f927f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:32.224092 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa48f4826ba44e99169dce15e0a0760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:32.223821 32487 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa48f4826ba44e99169dce15e0a0760 (DEFAULT_TABLE table=test-table [id=b9b51ebce45c4e55b2029079f6f927f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:07:32.224272 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa48f4826ba44e99169dce15e0a0760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:32.224761 32487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa48f4826ba44e99169dce15e0a0760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:07:32.230698 370 tablet_bootstrap.cc:492] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: Bootstrap starting.
I20260928 21:07:32.231907 372 tablet_bootstrap.cc:492] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d: Bootstrap starting.
I20260928 21:07:32.232264 371 tablet_bootstrap.cc:492] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Bootstrap starting.
I20260928 21:07:32.234094 370 tablet_bootstrap.cc:654] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:32.234100 372 tablet_bootstrap.cc:654] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:32.234421 371 tablet_bootstrap.cc:654] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Neither blocks nor log segments found. Creating new log.
I20260928 21:07:32.235075 370 log.cc:824] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:32.235107 371 log.cc:824] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:32.235270 372 log.cc:824] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:07:32.237113 371 tablet_bootstrap.cc:492] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: No bootstrap required, opened a new log
I20260928 21:07:32.237321 371 ts_tablet_manager.cc:1397] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 21:07:32.237684 370 tablet_bootstrap.cc:492] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: No bootstrap required, opened a new log
I20260928 21:07:32.237874 370 ts_tablet_manager.cc:1397] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:07:32.240238 371 raft_consensus.cc:348] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.240501 371 raft_consensus.cc:374] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:32.240566 371 raft_consensus.cc:729] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72765ac170da47d685530f22b0017343, State: Initialized, Role: FOLLOWER
I20260928 21:07:32.240599 370 raft_consensus.cc:348] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.240895 370 raft_consensus.cc:374] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:32.240958 370 raft_consensus.cc:729] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcd3660d07f440a9fa279503597865d, State: Initialized, Role: FOLLOWER
I20260928 21:07:32.241134 371 consensus_queue.cc:260] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.241376 370 consensus_queue.cc:260] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.242002 353 heartbeater.cc:503] Master 127.21.140.126:36507 was elected leader, sending a full tablet report...
I20260928 21:07:32.242326 371 ts_tablet_manager.cc:1428] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 21:07:32.242185 372 tablet_bootstrap.cc:492] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d: No bootstrap required, opened a new log
I20260928 21:07:32.242673 32687 heartbeater.cc:503] Master 127.21.140.126:36507 was elected leader, sending a full tablet report...
I20260928 21:07:32.242955 372 ts_tablet_manager.cc:1397] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260928 21:07:32.243822 370 ts_tablet_manager.cc:1428] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:07:32.246711 372 raft_consensus.cc:348] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.247028 372 raft_consensus.cc:374] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:07:32.247082 372 raft_consensus.cc:729] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6318aede454955800d8365da8bb70d, State: Initialized, Role: FOLLOWER
I20260928 21:07:32.247553 372 consensus_queue.cc:260] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.248206 32553 heartbeater.cc:503] Master 127.21.140.126:36507 was elected leader, sending a full tablet report...
I20260928 21:07:32.248467 372 ts_tablet_manager.cc:1428] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:07:32.378686 376 raft_consensus.cc:482] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:07:32.378932 376 raft_consensus.cc:504] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.380296 376 leader_election.cc:288] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da6318aede454955800d8365da8bb70d (127.21.140.65:37729), efcd3660d07f440a9fa279503597865d (127.21.140.66:40211)
I20260928 21:07:32.385388 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.380736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:46625 (local address 127.21.140.65:37729)
0928 21:07:32.380911 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.380915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.380935 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:07:32.381043 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.381049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.381107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.381197 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:32.381534 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.382352 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.383997 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.384645 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.384785 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.384847 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.385178 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.385261 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 21:07:32.385388 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.380723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:43247 (local address 127.21.140.66:40211)
0928 21:07:32.380882 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.380886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.381395 (+ 509us) server_negotiation.cc:407] Connection header received
0928 21:07:32.381550 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.381553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.381610 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.381694 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:07:32.382049 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.382685 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.384114 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.384738 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.384841 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.384894 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.385181 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.385262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:07:32.385360 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.380601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:37729 (local address 127.21.140.67:46625)
0928 21:07:32.380744 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.380763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:32.380896 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:32.381233 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:32.381237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:32.381257 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:32.381398 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.381405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.382502 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.382505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.383805 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.383816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.384803 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.384808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.385004 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.385032 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:07:32.385100 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:07:32.385193 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":84}
I20260928 21:07:32.385367 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.380601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:40211 (local address 127.21.140.67:43247)
0928 21:07:32.381285 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.381299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:07:32.381424 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:07:32.381742 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:07:32.381745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:07:32.381764 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:07:32.381873 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.381877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.382809 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.382813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.383993 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:07:32.384000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.384838 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.384841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:07:32.384936 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.384987 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 21:07:32.385071 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:07:32.385181 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":306,"thread_start_us":135,"threads_started":1}
I20260928 21:07:32.386241 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa48f4826ba44e99169dce15e0a0760" candidate_uuid: "72765ac170da47d685530f22b0017343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6318aede454955800d8365da8bb70d" is_pre_election: true
I20260928 21:07:32.386281 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa48f4826ba44e99169dce15e0a0760" candidate_uuid: "72765ac170da47d685530f22b0017343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcd3660d07f440a9fa279503597865d" is_pre_election: true
I20260928 21:07:32.386602 32507 raft_consensus.cc:2454] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72765ac170da47d685530f22b0017343 in term 0.
I20260928 21:07:32.386602 32641 raft_consensus.cc:2454] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72765ac170da47d685530f22b0017343 in term 0.
I20260928 21:07:32.387130 32710 leader_election.cc:302] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72765ac170da47d685530f22b0017343, efcd3660d07f440a9fa279503597865d; no voters:
I20260928 21:07:32.387449 376 raft_consensus.cc:2781] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:07:32.387573 376 raft_consensus.cc:482] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:07:32.387650 376 raft_consensus.cc:3037] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:32.388855 376 raft_consensus.cc:504] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.389281 376 leader_election.cc:288] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [CANDIDATE]: Term 1 election: Requested vote from peers da6318aede454955800d8365da8bb70d (127.21.140.65:37729), efcd3660d07f440a9fa279503597865d (127.21.140.66:40211)
I20260928 21:07:32.389683 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa48f4826ba44e99169dce15e0a0760" candidate_uuid: "72765ac170da47d685530f22b0017343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6318aede454955800d8365da8bb70d"
I20260928 21:07:32.389706 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa48f4826ba44e99169dce15e0a0760" candidate_uuid: "72765ac170da47d685530f22b0017343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efcd3660d07f440a9fa279503597865d"
I20260928 21:07:32.389845 32641 raft_consensus.cc:3037] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:32.389835 32507 raft_consensus.cc:3037] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:07:32.390949 32641 raft_consensus.cc:2454] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72765ac170da47d685530f22b0017343 in term 1.
I20260928 21:07:32.391033 32507 raft_consensus.cc:2454] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72765ac170da47d685530f22b0017343 in term 1.
I20260928 21:07:32.391288 32710 leader_election.cc:302] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72765ac170da47d685530f22b0017343, efcd3660d07f440a9fa279503597865d; no voters:
I20260928 21:07:32.391456 376 raft_consensus.cc:2781] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:07:32.391692 376 raft_consensus.cc:686] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 1 LEADER]: Becoming Leader. State: Replica: 72765ac170da47d685530f22b0017343, State: Running, Role: LEADER
I20260928 21:07:32.392069 376 consensus_queue.cc:237] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } }
I20260928 21:07:32.395677 32361 catalog_manager.cc:5495] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72765ac170da47d685530f22b0017343 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "72765ac170da47d685530f22b0017343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72765ac170da47d685530f22b0017343" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 } health_report { overall_health: UNKNOWN } } }
I20260928 21:07:32.409822 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.406105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.21.140.67:33769)
0928 21:07:32.406259 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:07:32.406264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:07:32.406278 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:07:32.406473 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:07:32.406477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:07:32.406573 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:07:32.406697 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:07:32.406938 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.407448 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.408335 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:07:32.408555 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:07:32.408668 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:07:32.408974 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:07:32.409393 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:07:32.409529 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:07:32.409590 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:07:32.409648 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:07:32.412240 354 tablet.cc:2369] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:07:32.415988 32641 raft_consensus.cc:1261] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 1 FOLLOWER]: Refusing update from remote peer 72765ac170da47d685530f22b0017343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:32.415989 32507 raft_consensus.cc:1261] T 0fa48f4826ba44e99169dce15e0a0760 P da6318aede454955800d8365da8bb70d [term 1 FOLLOWER]: Refusing update from remote peer 72765ac170da47d685530f22b0017343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:07:32.416762 376 consensus_queue.cc:1035] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6318aede454955800d8365da8bb70d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 37729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:32.417001 380 consensus_queue.cc:1035] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efcd3660d07f440a9fa279503597865d" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:07:32.437005 32361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:32.437172 32361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:07:32.439709 32361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52e2bcdbf97748b689ffbc71404b3c7b: Sending DeleteTablet for 3 replicas of tablet 0fa48f4826ba44e99169dce15e0a0760
I20260928 21:07:32.440538 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa48f4826ba44e99169dce15e0a0760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:32 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:41798
I20260928 21:07:32.441159 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 32424
I20260928 21:07:32.441159 32621 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa48f4826ba44e99169dce15e0a0760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:07:32 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:35844
I20260928 21:07:32.443478 394 tablet_replica.cc:330] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d: stopping tablet replica
I20260928 21:07:32.443804 394 raft_consensus.cc:2229] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:07:32.444108 394 raft_consensus.cc:2258] T 0fa48f4826ba44e99169dce15e0a0760 P efcd3660d07f440a9fa279503597865d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:32.444439 393 tablet_replica.cc:330] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: stopping tablet replica
I20260928 21:07:32.444761 393 raft_consensus.cc:2229] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:07:32.445111 393 raft_consensus.cc:2258] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:07:32.446496 393 ts_tablet_manager.cc:1905] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:07:32.449101 393 ts_tablet_manager.cc:1918] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:07:32.449198 393 log.cc:1196] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/0fa48f4826ba44e99169dce15e0a0760
I20260928 21:07:32.449502 393 ts_tablet_manager.cc:1939] T 0fa48f4826ba44e99169dce15e0a0760 P 72765ac170da47d685530f22b0017343: Deleting consensus metadata
I20260928 21:07:32.450771 32346 catalog_manager.cc:4841] TS 72765ac170da47d685530f22b0017343 (127.21.140.67:33769): tablet 0fa48f4826ba44e99169dce15e0a0760 (table test-table [id=b9b51ebce45c4e55b2029079f6f927f4]) successfully deleted
W20260928 21:07:32.451613 32347 connection.cc:535] client connection to 127.21.140.65:37729 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:07:32.451761 32347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:07:32.451887 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 32558
I20260928 21:07:32.458112 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.457738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:37729 (local address 127.0.0.1:38650)
0928 21:07:32.457913 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.458018 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 21:07:32.458423 32347 catalog_manager.cc:4584] TS da6318aede454955800d8365da8bb70d (127.21.140.65:37729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fa48f4826ba44e99169dce15e0a0760: Network error: Client connection negotiation failed: client connection to 127.21.140.65:37729: connect: Connection refused (error 111)
I20260928 21:07:32.460052 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 32692
I20260928 21:07:32.460927 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:07:32.460214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:40211 (local address 127.0.0.1:35848)
0928 21:07:32.460753 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 21:07:32.460835 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:40211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
I20260928 21:07:32.465464 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 32330
2026-09-28T21:07:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2412 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:07:32 dist-test-slave-d5wc krb5kdc[398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:07:32 dist-test-slave-d5wc krb5kdc[398](info): set up 2 sockets
Sep 28 21:07:32 dist-test-slave-d5wc krb5kdc[398](info): commencing operation
krb5kdc: starting...
W20260928 21:07:33.513659 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:07:33 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:07:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/rangeradmin_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/rangeradmin_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/rangerlookup_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/rangerlookup_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
I20260928 21:07:33.696310 22065 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskczrM8B/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/postgres -l logfile start
2026-09-28 21:07:36.461 UTC [430] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:07:36.461 UTC [430] LOG: listening on IPv4 address "127.21.140.84", port 58461
2026-09-28 21:07:36.465 UTC [430] LOG: listening on Unix socket "/tmp/.s.PGSQL.58461"
2026-09-28 21:07:36.493 UTC [433] LOG: database system was shut down at 2026-09-28 21:07:34 UTC
2026-09-28 21:07:36.497 UTC [430] LOG: database system is ready to accept connections
I20260928 21:07:37.456375 22065 mini_postgres.cc:96] Postgres bound to 58461
2026-09-28 21:07:37.463 UTC [441] FATAL: database "slave" does not exist
127.21.140.84:58461 - accepting connections
I20260928 21:07:37.464229 22065 mini_ranger.cc:161] Starting Ranger...
I20260928 21:07:37.479920 22065 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:07:37.982781 22065 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:07:37.982882 22065 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-admin
2026-09-28 21:07:38,286 [I] DB FLAVOR :POSTGRES
2026-09-28 21:07:38,287 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:07:38,287 [I] Checking connection..
2026-09-28 21:07:38,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:07:38,538 [I] Checking connection passed.
2026-09-28 21:07:38,538 [I] --------- Verifying version history table ---------
2026-09-28 21:07:38,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:07:38,797 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:07:38,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:07:39,042 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:07:39,043 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:07:39,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:07:39.292 UTC [505] WARNING: there is no transaction in progress
2026-09-28 21:07:39,298 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:07:39,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:07:39,540 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:07:39,540 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:07:39,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:07:39,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:07:40,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d5wc.c.gcp-upstream.internal','N') ;"
2026-09-28 21:07:40,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:07:40,509 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:07:40,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:07:40,802 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:07:40,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:07:41,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:07:41,294 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:07:41,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:07:42.279 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.296 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.314 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.329 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.346 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.363 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.420 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.427 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.442 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.452 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.463 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.477 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.488 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.496 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.504 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:42.511 UTC [614] WARNING: there is no transaction in progress
2026-09-28 21:07:43,156 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:07:43,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:07:43,415 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:07:43,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:07:43,682 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:07:43,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:07:43,911 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:07:43,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:07:44,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:07:44,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d5wc.c.gcp-upstream.internal';"
2026-09-28 21:07:44,653 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:07:44,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:07:44,915 [I] DB_PATCHES have already been applied
I20260928 21:07:44.921279 22065 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-admin:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskczrM8B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:07:44.921357 22065 mini_ranger.cc:193] Using host: 127.21.140.84
I20260928 21:07:44.924834 22065 mini_ranger.cc:227] Ranger admin URL: http://127.21.140.84:53245
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:07:46.042519 22065 mini_ranger.cc:160] Time spent starting Ranger: real 8.578s user 0.001s sys 0.009s
Sep 28, 2026 9:08:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:08:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 9:08:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:08:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:08:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:08:07.540874 22065 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/rangerkms_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/rangerkms_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.140.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.140.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:08:07 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629687, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:08:07.637373 22065 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:08:07.646891 22065 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:08:08.546046 22065 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:08:08.546144 22065 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms
I20260928 21:08:08.546247 22065 mini_ranger_kms.cc:227] Using postgres at 127.21.140.84:58461
2026-09-28 21:08:08,763 [I] DB FLAVOR :POSTGRES
2026-09-28 21:08:08,763 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:08:08,763 [I] Checking connection
2026-09-28 21:08:08,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:08:09,028 [I] connection success
2026-09-28 21:08:09,028 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:08:09,028 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:08:09,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:08:09,266 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:08:09,267 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:08:09,267 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:08:09,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:58461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:08:09,568 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:08:10.087109 22065 mini_ranger_kms.cc:310] Created kms service
I20260928 21:08:10.266429 22065 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:08:10.314553 22065 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:08:10 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629687, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:08:10.700870 22065 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:08:10.701026 22065 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms:/tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:08:10.701048 22065 mini_ranger_kms.cc:240] Using host: 127.21.140.84
I20260928 21:08:10.705013 22065 mini_ranger_kms.cc:276] Ranger KMS PID: 796
I20260928 21:08:10.705150 22065 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.140.84:34331
Sep 28, 2026 9:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:08:13.020270 22065 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.383s user 0.002s sys 0.199s
I20260928 21:08:13.020424 22065 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:08:13.020543 22065 mini_ranger_kms.cc:401] 127.21.140.84:34331/kms/v1/keys
Sep 28 21:08:13 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790629693, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.140.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:08:13 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629693, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.140.84@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
Sep 28 21:08:13 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629693, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.140.84@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
Sep 28 21:08:16 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:08:16.142831 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:41793
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:41793
--ranger_config_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:34331/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:08:16.258255 848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:08:16.258510 848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:08:16.258549 848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:08:16.262486 848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:08:16.262542 848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:08:16.262562 848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:08:16.262578 848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:08:16.262598 848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:08:16.267894 848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.140.84:34331/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:41793
--ranger_config_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:08:16.268915 848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:08:16.269866 848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:08:16.278527 853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:16.278617 848 server_base.cc:1034] running on GCE node
W20260928 21:08:16.278630 854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:08:16.278527 856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:16.279345 848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:08:16.280373 848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:08:16.281560 848 hybrid_clock.cc:648] HybridClock initialized: now 1790629696281531 us; error 44 us; skew 500 ppm
Sep 28 21:08:16 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629696, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:08:16.289175 848 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:08:16.290246 848 webserver.cc:460] Webserver started at http://127.21.140.126:33657/ using document root <none> and password file <none>
I20260928 21:08:16.290825 848 fs_manager.cc:359] Metadata directory not provided
I20260928 21:08:16.290884 848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:08:16.291115 848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:08:16 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629696, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:08:16.345914 848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "a5630d92d76449e79b5301383b65bad1"
format_stamp: "Formatted at 2026-09-28 21:08:16 on dist-test-slave-d5wc"
server_key: "74be65209eead97b7ffdbad3d8d2f5cc"
server_key_iv: "460a7dab3811f96bc2ee3c512a7eac35"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:16.346645 848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "a5630d92d76449e79b5301383b65bad1"
format_stamp: "Formatted at 2026-09-28 21:08:16 on dist-test-slave-d5wc"
server_key: "74be65209eead97b7ffdbad3d8d2f5cc"
server_key_iv: "460a7dab3811f96bc2ee3c512a7eac35"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:16.350919 848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:08:16.364760 863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:16.366256 848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:08:16.366398 848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "a5630d92d76449e79b5301383b65bad1"
format_stamp: "Formatted at 2026-09-28 21:08:16 on dist-test-slave-d5wc"
server_key: "74be65209eead97b7ffdbad3d8d2f5cc"
server_key_iv: "460a7dab3811f96bc2ee3c512a7eac35"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:16.366542 848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:08:16.386396 848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:08:16.389230 848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:08:16.389496 848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:08:16.396958 848 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:41793
I20260928 21:08:16.396973 915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:41793 every 8 connection(s)
I20260928 21:08:16.397945 848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:08:16.400554 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 848
I20260928 21:08:16.400655 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:08:16.400818 916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:08:16.401010 22065 external_mini_cluster.cc:1428] Setting key 5e944f0ab4c0f35155d790f9f2f8dfe6
I20260928 21:08:16.408453 916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1: Bootstrap starting.
Sep 28 21:08:16 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:08:16.411015 916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1: Neither blocks nor log segments found. Creating new log.
I20260928 21:08:16.412106 916 log.cc:824] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:08:16.414619 916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1: No bootstrap required, opened a new log
I20260928 21:08:16.417225 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:16.402588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.21.140.126:41793)
0928 21:08:16.403015 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:16.403032 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:16.403073 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:08:16.403815 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:16.403838 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:16.404254 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:16.404639 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:08:16.405358 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:16.406959 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:16.407969 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:16.408650 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:16.408826 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:16.411510 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:08:16.411517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:08:16.411533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:08:16.411564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:08:16.414149 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:16.414631 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:16.414640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:16.414647 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:16.414801 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:16.415079 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:16.415082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:16.415084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:16.415574 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:08:16.415787 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:16.415992 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:08:16.416268 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260928 21:08:16.418597 916 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5630d92d76449e79b5301383b65bad1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 41793 } }
I20260928 21:08:16.418869 916 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:08:16.418938 916 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5630d92d76449e79b5301383b65bad1, State: Initialized, Role: FOLLOWER
I20260928 21:08:16.419363 916 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5630d92d76449e79b5301383b65bad1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 41793 } }
I20260928 21:08:16.419533 916 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:08:16.419636 916 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:08:16.419737 916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:08:16.420660 916 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5630d92d76449e79b5301383b65bad1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 41793 } }
I20260928 21:08:16.421031 916 leader_election.cc:302] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5630d92d76449e79b5301383b65bad1; no voters:
I20260928 21:08:16.421368 916 leader_election.cc:288] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:08:16.421514 921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:08:16.421813 921 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [term 1 LEADER]: Becoming Leader. State: Replica: a5630d92d76449e79b5301383b65bad1, State: Running, Role: LEADER
I20260928 21:08:16.422235 921 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5630d92d76449e79b5301383b65bad1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 41793 } }
I20260928 21:08:16.422679 916 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:08:16.424510 923 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5630d92d76449e79b5301383b65bad1. Latest consensus state: current_term: 1 leader_uuid: "a5630d92d76449e79b5301383b65bad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5630d92d76449e79b5301383b65bad1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 41793 } } }
I20260928 21:08:16.424633 923 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:08:16.424925 922 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5630d92d76449e79b5301383b65bad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5630d92d76449e79b5301383b65bad1" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 41793 } } }
I20260928 21:08:16.425009 922 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:08:16.425406 930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:08:16.425788 916 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:08:16.431335 930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:08:16.437060 930 catalog_manager.cc:1347] Generated new cluster ID: 32ceb830ea784a6c8b610ed42ca39f99
I20260928 21:08:16.437120 930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:08:16.459826 930 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:08:16.461228 930 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:08:16.469452 930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1: Generated new TSK 0
I20260928 21:08:16.470281 930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:08:17.150164 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:34331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:41793
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:08:17.288761 962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:08:17.289131 962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:08:17.289222 962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:08:17.294965 962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:08:17.295051 962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:08:17.295152 962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:08:17.301390 962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.140.84:34331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:08:17.302667 962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:08:17.303689 962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:08:17.313294 972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:08:17.313295 970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:08:17.313812 969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:17.314152 962 server_base.cc:1034] running on GCE node
I20260928 21:08:17.314599 962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:08:17.315295 962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:08:17.316475 962 hybrid_clock.cc:648] HybridClock initialized: now 1790629697316448 us; error 43 us; skew 500 ppm
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:08:17.319653 962 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:08:17.320657 962 webserver.cc:460] Webserver started at http://127.21.140.65:35131/ using document root <none> and password file <none>
I20260928 21:08:17.321274 962 fs_manager.cc:359] Metadata directory not provided
I20260928 21:08:17.321376 962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:08:17.321609 962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:08:17.336840 962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "f890a1283f9244baa69d25fbb66628c5"
format_stamp: "Formatted at 2026-09-28 21:08:17 on dist-test-slave-d5wc"
server_key: "272aaf7662eded61bc3e784ea200676f"
server_key_iv: "9ccbce34367ba41d508c4969c262efdd"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:17.337568 962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "f890a1283f9244baa69d25fbb66628c5"
format_stamp: "Formatted at 2026-09-28 21:08:17 on dist-test-slave-d5wc"
server_key: "272aaf7662eded61bc3e784ea200676f"
server_key_iv: "9ccbce34367ba41d508c4969c262efdd"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:17.342813 962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 21:08:17.361359 979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:17.367384 962 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260928 21:08:17.367597 962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "f890a1283f9244baa69d25fbb66628c5"
format_stamp: "Formatted at 2026-09-28 21:08:17 on dist-test-slave-d5wc"
server_key: "272aaf7662eded61bc3e784ea200676f"
server_key_iv: "9ccbce34367ba41d508c4969c262efdd"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:17.367810 962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:08:17.391198 962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:08:17.395085 962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:08:17.395371 962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:08:17.396062 962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:08:17.397075 962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:08:17.397164 962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:17.397230 962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:08:17.397264 962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:17.407531 962 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:42221
I20260928 21:08:17.408692 962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:08:17.410059 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 962
I20260928 21:08:17.410185 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:08:17.410650 22065 external_mini_cluster.cc:1428] Setting key 0d00855c48c7c74b96145264882a4d45
I20260928 21:08:17.410277 1093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:42221 every 8 connection(s)
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:08:17.442440 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:17.412120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:54377 (local address 127.21.140.126:41793)
0928 21:08:17.417516 (+ 5396us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:17.417523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:17.419082 (+ 1559us) server_negotiation.cc:407] Connection header received
0928 21:08:17.419711 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:17.419714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:17.419772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:17.419848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:08:17.421159 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.422027 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.425388 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.425852 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.425954 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:17.437439 (+ 11485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:08:17.437443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:08:17.437446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:08:17.437481 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:08:17.439496 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:17.440177 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:17.440181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:17.440183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:17.440275 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:17.440736 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:17.440739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:17.440741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:17.441038 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:08:17.441136 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:17.442090 (+ 954us) server_negotiation.cc:299] Negotiation successful
0928 21:08:17.442229 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5314,"thread_start_us":129,"threads_started":1}
I20260928 21:08:17.444229 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:17.415174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:41793 (local address 127.21.140.65:54377)
0928 21:08:17.418352 (+ 3178us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:17.418424 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:17.419468 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:17.420019 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:17.420026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:17.420361 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:08:17.421000 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:17.421015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.422208 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.422212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:17.425205 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:17.425220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.434153 (+ 8933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.434159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:17.434441 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:17.435331 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:08:17.435367 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:08:17.437164 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:08:17.439645 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:08:17.439651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:08:17.439671 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:08:17.440040 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:08:17.440385 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:08:17.440389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:08:17.440391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:08:17.440598 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:08:17.441191 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:08:17.441199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:08:17.441535 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 21:08:17.441740 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 21:08:17.443083 (+ 1343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2910,"thread_start_us":166,"threads_started":1}
I20260928 21:08:17.445946 1094 heartbeater.cc:347] Connected to a master server at 127.21.140.126:41793
I20260928 21:08:17.446285 1094 heartbeater.cc:464] Registering TS with master...
I20260928 21:08:17.446916 1094 heartbeater.cc:511] Master 127.21.140.126:41793 requested a full tablet report, sending...
I20260928 21:08:17.448829 880 ts_manager.cc:194] Registered new tserver with Master: f890a1283f9244baa69d25fbb66628c5 (127.21.140.65:42221)
I20260928 21:08:17.450438 880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:54377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:08:17.461720 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:34331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:41793
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:08:17.617112 1101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:08:17.617529 1101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:08:17.617597 1101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:08:17.624377 1101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:08:17.624461 1101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:08:17.624547 1101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:08:17.632498 1101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.140.84:34331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:08:17.633792 1101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:08:17.635092 1101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:08:17.647140 1110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:08:17.647528 1107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:17.648419 1101 server_base.cc:1034] running on GCE node
W20260928 21:08:17.649248 1108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:17.649741 1101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:08:17.650511 1101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:08:17.651706 1101 hybrid_clock.cc:648] HybridClock initialized: now 1790629697651680 us; error 44 us; skew 500 ppm
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:08:17.655230 1101 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:08:17.656154 1101 webserver.cc:460] Webserver started at http://127.21.140.66:37141/ using document root <none> and password file <none>
I20260928 21:08:17.656720 1101 fs_manager.cc:359] Metadata directory not provided
I20260928 21:08:17.656780 1101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:08:17.656986 1101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:08:17.672771 1101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "a063582ce58c4c53a46be4c81144e717"
format_stamp: "Formatted at 2026-09-28 21:08:17 on dist-test-slave-d5wc"
server_key: "12b30ddb09b443008b9ab2faff0ee1cb"
server_key_iv: "932b83e913a311120a19d0257d7a92d9"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:17.673627 1101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "a063582ce58c4c53a46be4c81144e717"
format_stamp: "Formatted at 2026-09-28 21:08:17 on dist-test-slave-d5wc"
server_key: "12b30ddb09b443008b9ab2faff0ee1cb"
server_key_iv: "932b83e913a311120a19d0257d7a92d9"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:17.678687 1101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 21:08:17.707309 1117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:17.710873 1101 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 21:08:17.711095 1101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "a063582ce58c4c53a46be4c81144e717"
format_stamp: "Formatted at 2026-09-28 21:08:17 on dist-test-slave-d5wc"
server_key: "12b30ddb09b443008b9ab2faff0ee1cb"
server_key_iv: "932b83e913a311120a19d0257d7a92d9"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:17.711354 1101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:08:17.733529 1101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:08:17.749857 1101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:08:17.750212 1101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:08:17.751020 1101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:08:17.752190 1101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:08:17.756129 1101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 21:08:17.756313 1101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:08:17.756494 1101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:17.769447 1101 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:45475
I20260928 21:08:17.770828 1101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:08:17.771202 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 1101
I20260928 21:08:17.771327 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:08:17.771658 22065 external_mini_cluster.cc:1428] Setting key 389927f1239e692aa1b098d0d524cbe1
I20260928 21:08:17.783429 1233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:45475 every 8 connection(s)
Sep 28 21:08:17 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:08:17.799553 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:17.782946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:38401 (local address 127.21.140.126:41793)
0928 21:08:17.786683 (+ 3737us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:17.786688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:17.786706 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:08:17.786756 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:17.786759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:17.786807 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:17.786891 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:08:17.788879 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.789695 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.791458 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.791816 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.791945 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:17.795148 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:08:17.795152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:08:17.795155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:08:17.795183 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:08:17.796979 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:17.797723 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:17.797727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:17.797729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:17.797818 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:17.798258 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:17.798261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:17.798263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:17.798487 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:08:17.798585 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:17.799255 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 21:08:17.799390 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3628}
I20260928 21:08:17.800738 1237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:17.781708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:41793 (local address 127.21.140.66:38401)
0928 21:08:17.782388 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:17.782454 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:17.783561 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:17.787503 (+ 3942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:17.787514 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:17.787953 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:08:17.788665 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:17.788686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.789860 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.789862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:17.791307 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:17.791319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:17.792137 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:17.792141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:17.792341 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:17.793020 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:08:17.793042 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:08:17.794932 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:08:17.797141 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:08:17.797147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:08:17.797164 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:08:17.797581 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:08:17.797927 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:08:17.797931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:08:17.797933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:08:17.798144 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:08:17.798605 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:08:17.798612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:08:17.798976 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 21:08:17.799191 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 21:08:17.799573 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":198,"threads_started":1}
I20260928 21:08:17.802125 1235 heartbeater.cc:347] Connected to a master server at 127.21.140.126:41793
I20260928 21:08:17.802402 1235 heartbeater.cc:464] Registering TS with master...
I20260928 21:08:17.803016 1235 heartbeater.cc:511] Master 127.21.140.126:41793 requested a full tablet report, sending...
I20260928 21:08:17.804210 880 ts_manager.cc:194] Registered new tserver with Master: a063582ce58c4c53a46be4c81144e717 (127.21.140.66:45475)
I20260928 21:08:17.804996 880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:38401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:08:17.842406 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:34331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:41793
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:08:18.015861 1239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:08:18.016733 1239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:08:18.016881 1239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:08:18.023483 1239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:08:18.023630 1239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:08:18.023818 1239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:08:18.032353 1239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.140.84:34331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:08:18.033955 1239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:08:18.035344 1239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:08:18.048488 1244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:08:18.048472 1245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:18.050064 1239 server_base.cc:1034] running on GCE node
W20260928 21:08:18.050422 1247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:08:18.050951 1239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:08:18.051759 1239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:08:18.052922 1239 hybrid_clock.cc:648] HybridClock initialized: now 1790629698052842 us; error 92 us; skew 500 ppm
Sep 28 21:08:18 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
Sep 28 21:08:18 dist-test-slave-d5wc krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629698, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:08:18.056896 1239 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:08:18.058037 1239 webserver.cc:460] Webserver started at http://127.21.140.67:34341/ using document root <none> and password file <none>
I20260928 21:08:18.058874 1239 fs_manager.cc:359] Metadata directory not provided
I20260928 21:08:18.059018 1239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:08:18.059283 1239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:08:18 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629698, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:08:18.080864 1239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "6544972a0def47fd9fcd2fa8e743e12d"
format_stamp: "Formatted at 2026-09-28 21:08:18 on dist-test-slave-d5wc"
server_key: "0fbb45922d1bd6b079188d539099103d"
server_key_iv: "7fe139c72d544207813a164079063dfe"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:18.081712 1239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "6544972a0def47fd9fcd2fa8e743e12d"
format_stamp: "Formatted at 2026-09-28 21:08:18 on dist-test-slave-d5wc"
server_key: "0fbb45922d1bd6b079188d539099103d"
server_key_iv: "7fe139c72d544207813a164079063dfe"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:18.096992 1239 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.001s sys 0.003s
I20260928 21:08:18.129717 1254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:18.131528 1239 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 21:08:18.131743 1239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "6544972a0def47fd9fcd2fa8e743e12d"
format_stamp: "Formatted at 2026-09-28 21:08:18 on dist-test-slave-d5wc"
server_key: "0fbb45922d1bd6b079188d539099103d"
server_key_iv: "7fe139c72d544207813a164079063dfe"
server_key_version: "kuduclusterkey@0"
I20260928 21:08:18.131976 1239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:08:18.149123 1239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:08:18.153025 1239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:08:18.153283 1239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:08:18.153999 1239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:08:18.155059 1239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:08:18.155165 1239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:18.155272 1239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:08:18.155333 1239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:08:18.166402 1239 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:36337
I20260928 21:08:18.167727 1239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:08:18.167919 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 1239
I20260928 21:08:18.168051 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:08:18.168349 22065 external_mini_cluster.cc:1428] Setting key 25916fb80731fc9a5332a779bab33a17
I20260928 21:08:18.176621 1368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:36337 every 8 connection(s)
Sep 28 21:08:18 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629698, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:08:18.189374 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.173788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:49099 (local address 127.21.140.126:41793)
0928 21:08:18.173899 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.173903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.174395 (+ 492us) server_negotiation.cc:407] Connection header received
0928 21:08:18.175791 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.175795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.175857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.175918 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:08:18.177578 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.178434 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.180095 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.180473 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.180551 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.185027 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:08:18.185031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:08:18.185034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:08:18.185063 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:08:18.186936 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:18.187601 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:18.187604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:18.187606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:18.187689 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:18.188091 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:18.188094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:18.188096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:18.188316 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:08:18.188387 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.189069 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.189217 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:08:18.191296 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.172700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:41793 (local address 127.21.140.67:49099)
0928 21:08:18.174238 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.174300 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.175526 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.176353 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:18.176364 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:18.176743 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:08:18.177378 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.177397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.178604 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.178607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.179954 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.179965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.180819 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.180822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.181012 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.183051 (+ 2039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:08:18.183071 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:08:18.184837 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:08:18.187108 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:08:18.187113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:08:18.187129 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:08:18.187484 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:08:18.187810 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:08:18.187813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:08:18.187815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:08:18.187986 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:08:18.188468 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:08:18.188475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:08:18.188790 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 21:08:18.189845 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0928 21:08:18.190238 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"thread_start_us":202,"threads_started":1}
I20260928 21:08:18.192873 1369 heartbeater.cc:347] Connected to a master server at 127.21.140.126:41793
I20260928 21:08:18.193126 1369 heartbeater.cc:464] Registering TS with master...
I20260928 21:08:18.193727 1369 heartbeater.cc:511] Master 127.21.140.126:41793 requested a full tablet report, sending...
I20260928 21:08:18.195204 880 ts_manager.cc:194] Registered new tserver with Master: 6544972a0def47fd9fcd2fa8e743e12d (127.21.140.67:36337)
I20260928 21:08:18.195905 880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:49099
I20260928 21:08:18.196765 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:08:18.212423 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.202460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.21.140.126:41793)
0928 21:08:18.202579 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.202584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.203230 (+ 646us) server_negotiation.cc:407] Connection header received
0928 21:08:18.203430 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.203433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.203484 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.203538 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:08:18.204542 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.205285 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.206459 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.206759 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.206839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.208737 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:08:18.208741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:08:18.208743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:08:18.208771 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:08:18.210540 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:18.211042 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:18.211046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:18.211048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:18.211133 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:08:18.211457 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:08:18.211460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:08:18.211462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:08:18.211670 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:08:18.211776 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.212120 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.212255 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:08:18.216256 880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:08:18.221163 880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:08:18.250435 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.232418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:42221 (local address 127.0.0.1:56660)
0928 21:08:18.237211 (+ 4793us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.237269 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.237398 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.238489 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:18.238496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:18.238521 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:08:18.238666 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.238674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.241145 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.241149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.243362 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.243373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.249806 (+ 6433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.249822 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.249951 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.250021 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 21:08:18.250140 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:08:18.250279 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4597,"thread_start_us":89,"threads_started":1}
I20260928 21:08:18.254688 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.234603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:36337 (local address 127.0.0.1:36834)
0928 21:08:18.243588 (+ 8985us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.243602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.243683 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.244731 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:18.244734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:18.244747 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:08:18.244865 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.244871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.247148 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.247151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.248417 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.248428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.254247 (+ 5819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.254250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.254354 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.254378 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:08:18.254451 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:08:18.254533 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8916,"thread_start_us":81,"threads_started":1}
I20260928 21:08:18.258809 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.233207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.21.140.65:42221)
0928 21:08:18.237760 (+ 4553us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.237767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.237791 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:08:18.237880 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.237885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.238143 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.238303 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:08:18.239841 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.241006 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.248580 (+ 7574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.249477 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.249592 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.249741 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.258534 (+ 8793us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.258661 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4441,"thread_start_us":160,"threads_started":1}
I20260928 21:08:18.259469 1383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.234230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.21.140.67:36337)
0928 21:08:18.239750 (+ 5520us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.239768 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.243778 (+ 4010us) server_negotiation.cc:407] Connection header received
0928 21:08:18.243880 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.243886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.244133 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.244298 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:08:18.244998 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.247012 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.252197 (+ 5185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.253075 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.253199 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.253356 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.259207 (+ 5851us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.259326 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4633,"thread_start_us":153,"threads_started":1}
I20260928 21:08:18.263032 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.235242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45475 (local address 127.0.0.1:49350)
0928 21:08:18.244542 (+ 9300us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.244557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.244644 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.254010 (+ 9366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:18.254013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:18.254033 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:08:18.254122 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.254128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.257135 (+ 3007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.257139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.258346 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.258357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.262597 (+ 4240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.262601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.262698 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.262724 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:08:18.262798 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:08:18.262893 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9233,"thread_start_us":81,"threads_started":1}
I20260928 21:08:18.262928 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 954f1e9e73a94a31bc685717404e118e (DEFAULT_TABLE table=test-table [id=af0e94042c0c4e6f84928b7ecb74a477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:08:18.264146 1027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954f1e9e73a94a31bc685717404e118e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:08:18.264566 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.235631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.21.140.66:45475)
0928 21:08:18.251910 (+ 16279us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.251928 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.251955 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:08:18.252035 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.252041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.253626 (+ 1585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.253797 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:08:18.254947 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.256139 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.261338 (+ 5199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.262230 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.262350 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.262524 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.264299 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.264410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16142,"thread_start_us":110,"threads_started":1}
I20260928 21:08:18.267455 1166 tablet_service.cc:1467] Processing CreateTablet for tablet 954f1e9e73a94a31bc685717404e118e (DEFAULT_TABLE table=test-table [id=af0e94042c0c4e6f84928b7ecb74a477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:08:18.267958 1302 tablet_service.cc:1467] Processing CreateTablet for tablet 954f1e9e73a94a31bc685717404e118e (DEFAULT_TABLE table=test-table [id=af0e94042c0c4e6f84928b7ecb74a477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:08:18.268568 1166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954f1e9e73a94a31bc685717404e118e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:08:18.268918 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954f1e9e73a94a31bc685717404e118e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:08:18.278278 1388 tablet_bootstrap.cc:492] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Bootstrap starting.
I20260928 21:08:18.281033 1388 tablet_bootstrap.cc:654] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Neither blocks nor log segments found. Creating new log.
I20260928 21:08:18.283506 1388 log.cc:824] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Log is configured to *not* fsync() on all Append() calls
I20260928 21:08:18.285722 1388 tablet_bootstrap.cc:492] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: No bootstrap required, opened a new log
I20260928 21:08:18.285940 1388 ts_tablet_manager.cc:1397] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 21:08:18.290089 1388 raft_consensus.cc:348] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.290385 1388 raft_consensus.cc:374] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:08:18.290452 1388 raft_consensus.cc:729] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a063582ce58c4c53a46be4c81144e717, State: Initialized, Role: FOLLOWER
I20260928 21:08:18.290648 1387 tablet_bootstrap.cc:492] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Bootstrap starting.
I20260928 21:08:18.290958 1388 consensus_queue.cc:260] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.292032 1388 ts_tablet_manager.cc:1428] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260928 21:08:18.292174 1390 tablet_bootstrap.cc:492] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: Bootstrap starting.
I20260928 21:08:18.292403 1235 heartbeater.cc:503] Master 127.21.140.126:41793 was elected leader, sending a full tablet report...
I20260928 21:08:18.299625 1390 tablet_bootstrap.cc:654] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: Neither blocks nor log segments found. Creating new log.
I20260928 21:08:18.302060 1387 tablet_bootstrap.cc:654] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Neither blocks nor log segments found. Creating new log.
I20260928 21:08:18.303194 1390 log.cc:824] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:08:18.304768 1387 log.cc:824] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:08:18.307449 1390 tablet_bootstrap.cc:492] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: No bootstrap required, opened a new log
I20260928 21:08:18.307758 1390 ts_tablet_manager.cc:1397] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20260928 21:08:18.314136 1390 raft_consensus.cc:348] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.314580 1390 raft_consensus.cc:374] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:08:18.314682 1390 raft_consensus.cc:729] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f890a1283f9244baa69d25fbb66628c5, State: Initialized, Role: FOLLOWER
I20260928 21:08:18.315310 1390 consensus_queue.cc:260] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.316852 1390 ts_tablet_manager.cc:1428] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260928 21:08:18.317312 1094 heartbeater.cc:503] Master 127.21.140.126:41793 was elected leader, sending a full tablet report...
I20260928 21:08:18.319398 1387 tablet_bootstrap.cc:492] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: No bootstrap required, opened a new log
I20260928 21:08:18.319635 1387 ts_tablet_manager.cc:1397] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20260928 21:08:18.324200 1387 raft_consensus.cc:348] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.324507 1387 raft_consensus.cc:374] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:08:18.324568 1387 raft_consensus.cc:729] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6544972a0def47fd9fcd2fa8e743e12d, State: Initialized, Role: FOLLOWER
I20260928 21:08:18.325078 1387 consensus_queue.cc:260] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.326195 1387 ts_tablet_manager.cc:1428] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 21:08:18.326583 1369 heartbeater.cc:503] Master 127.21.140.126:41793 was elected leader, sending a full tablet report...
I20260928 21:08:18.412676 1394 raft_consensus.cc:482] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:08:18.412883 1394 raft_consensus.cc:504] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.414181 1394 leader_election.cc:288] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6544972a0def47fd9fcd2fa8e743e12d (127.21.140.67:36337), a063582ce58c4c53a46be4c81144e717 (127.21.140.66:45475)
Sep 28 21:08:18 dist-test-slave-d5wc krb5kdc[398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629697, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
W20260928 21:08:18.417876 1095 tablet.cc:2369] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:08:18.418803 1370 tablet.cc:2369] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:08:18.427062 1383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.415206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:48929 (local address 127.21.140.67:36337)
0928 21:08:18.415463 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.415469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.421990 (+ 6521us) server_negotiation.cc:407] Connection header received
0928 21:08:18.422198 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.422201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.422247 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.422291 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:08:18.422641 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.423402 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.425542 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.426235 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.426338 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.426392 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.426847 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.426931 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260928 21:08:18.428439 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.414363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:36337 (local address 127.21.140.65:48929)
0928 21:08:18.421868 (+ 7505us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.421884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.422077 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.422421 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:18.422423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:18.422438 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:08:18.422532 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.422537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.423587 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.423590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.425002 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.425014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.426489 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.426493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.426588 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.426708 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 21:08:18.427270 (+ 562us) client_negotiation.cc:240] Negotiation successful
0928 21:08:18.427349 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7402,"thread_start_us":116,"threads_started":1}
I20260928 21:08:18.429221 1322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f1e9e73a94a31bc685717404e118e" candidate_uuid: "f890a1283f9244baa69d25fbb66628c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6544972a0def47fd9fcd2fa8e743e12d" is_pre_election: true
I20260928 21:08:18.429559 1322 raft_consensus.cc:2454] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f890a1283f9244baa69d25fbb66628c5 in term 0.
I20260928 21:08:18.430315 980 leader_election.cc:302] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6544972a0def47fd9fcd2fa8e743e12d, f890a1283f9244baa69d25fbb66628c5; no voters:
I20260928 21:08:18.430652 1394 raft_consensus.cc:2781] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:08:18.430744 1394 raft_consensus.cc:482] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:08:18.430787 1394 raft_consensus.cc:3037] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:08:18.432140 1394 raft_consensus.cc:504] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.432530 1394 leader_election.cc:288] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [CANDIDATE]: Term 1 election: Requested vote from peers 6544972a0def47fd9fcd2fa8e743e12d (127.21.140.67:36337), a063582ce58c4c53a46be4c81144e717 (127.21.140.66:45475)
I20260928 21:08:18.432955 1322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f1e9e73a94a31bc685717404e118e" candidate_uuid: "f890a1283f9244baa69d25fbb66628c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6544972a0def47fd9fcd2fa8e743e12d"
I20260928 21:08:18.433115 1322 raft_consensus.cc:3037] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:08:18.434505 1322 raft_consensus.cc:2454] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f890a1283f9244baa69d25fbb66628c5 in term 1.
I20260928 21:08:18.434903 980 leader_election.cc:302] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6544972a0def47fd9fcd2fa8e743e12d, f890a1283f9244baa69d25fbb66628c5; no voters:
I20260928 21:08:18.435096 1394 raft_consensus.cc:2781] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:08:18.435360 1394 raft_consensus.cc:686] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [term 1 LEADER]: Becoming Leader. State: Replica: f890a1283f9244baa69d25fbb66628c5, State: Running, Role: LEADER
I20260928 21:08:18.435729 1394 consensus_queue.cc:237] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } }
I20260928 21:08:18.439257 880 catalog_manager.cc:5495] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f890a1283f9244baa69d25fbb66628c5 (127.21.140.65). New cstate: current_term: 1 leader_uuid: "f890a1283f9244baa69d25fbb66628c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f890a1283f9244baa69d25fbb66628c5" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 42221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 } health_report { overall_health: UNKNOWN } } }
I20260928 21:08:18.446935 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.415321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:59721 (local address 127.21.140.66:45475)
0928 21:08:18.415462 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.415467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.425389 (+ 9922us) server_negotiation.cc:407] Connection header received
0928 21:08:18.425503 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.425507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.425566 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.425649 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:08:18.441625 (+ 15976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.442421 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.445759 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.446306 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.446449 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.446508 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.446706 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.446802 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:08:18.447628 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.414766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45475 (local address 127.21.140.65:59721)
0928 21:08:18.425298 (+ 10532us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.425312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.425411 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.438377 (+ 12966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:08:18.438380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:08:18.438403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:08:18.438517 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.438523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.442731 (+ 4208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.442735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.445604 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:08:18.445618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.446444 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.446447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:08:18.446549 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.446574 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:08:18.447392 (+ 818us) client_negotiation.cc:240] Negotiation successful
0928 21:08:18.447483 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10407,"thread_start_us":97,"threads_started":1}
I20260928 21:08:18.448302 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f1e9e73a94a31bc685717404e118e" candidate_uuid: "f890a1283f9244baa69d25fbb66628c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a063582ce58c4c53a46be4c81144e717"
I20260928 21:08:18.448570 1185 raft_consensus.cc:3037] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:08:18.448334 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f1e9e73a94a31bc685717404e118e" candidate_uuid: "f890a1283f9244baa69d25fbb66628c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a063582ce58c4c53a46be4c81144e717" is_pre_election: true
I20260928 21:08:18.449946 1185 raft_consensus.cc:2454] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f890a1283f9244baa69d25fbb66628c5 in term 1.
I20260928 21:08:18.479847 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.474316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.21.140.65:42221)
0928 21:08:18.474443 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:08:18.474448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:08:18.474668 (+ 220us) server_negotiation.cc:407] Connection header received
0928 21:08:18.474780 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:08:18.474783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:08:18.474856 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:08:18.474978 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:08:18.475295 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.476079 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.477314 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:08:18.477648 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:08:18.477747 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:08:18.478196 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:08:18.478821 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:08:18.478948 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:08:18.479597 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 21:08:18.479672 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:08:18.490901 1185 raft_consensus.cc:1261] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 1 FOLLOWER]: Refusing update from remote peer f890a1283f9244baa69d25fbb66628c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:08:18.490976 1322 raft_consensus.cc:1261] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 1 FOLLOWER]: Refusing update from remote peer f890a1283f9244baa69d25fbb66628c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:08:18.491753 1399 consensus_queue.cc:1035] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6544972a0def47fd9fcd2fa8e743e12d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:08:18.491943 1394 consensus_queue.cc:1035] T 954f1e9e73a94a31bc685717404e118e P f890a1283f9244baa69d25fbb66628c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a063582ce58c4c53a46be4c81144e717" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:08:18.516526 1404 mvcc.cc:205] Tried to move back new op lower bound from 7334419244998279168 to 7334419244795830272. Current Snapshot: MvccSnapshot[applied={T|T < 7334419244998279168}]
I20260928 21:08:18.518049 1405 mvcc.cc:205] Tried to move back new op lower bound from 7334419244998279168 to 7334419244795830272. Current Snapshot: MvccSnapshot[applied={T|T < 7334419244998279168}]
I20260928 21:08:18.528923 879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:08:18.529121 879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:08:18.532158 879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5630d92d76449e79b5301383b65bad1: Sending DeleteTablet for 3 replicas of tablet 954f1e9e73a94a31bc685717404e118e
I20260928 21:08:18.533677 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 954f1e9e73a94a31bc685717404e118e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:08:18 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:56660
I20260928 21:08:18.534297 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 954f1e9e73a94a31bc685717404e118e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:08:18 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:36834
I20260928 21:08:18.534787 1166 tablet_service.cc:1514] Processing DeleteTablet for tablet 954f1e9e73a94a31bc685717404e118e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:08:18 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:49350
I20260928 21:08:18.536051 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 962
I20260928 21:08:18.540647 1413 tablet_replica.cc:330] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: stopping tablet replica
I20260928 21:08:18.540941 1414 tablet_replica.cc:330] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: stopping tablet replica
I20260928 21:08:18.541378 1414 raft_consensus.cc:2229] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:08:18.541790 1413 raft_consensus.cc:2229] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:08:18.542236 1414 raft_consensus.cc:2258] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:08:18.542588 1413 raft_consensus.cc:2258] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:08:18.544457 1414 ts_tablet_manager.cc:1905] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:08:18.545469 1413 ts_tablet_manager.cc:1905] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:08:18.548362 1413 ts_tablet_manager.cc:1918] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:08:18.551163 1414 ts_tablet_manager.cc:1918] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:08:18.551334 1414 log.cc:1196] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/954f1e9e73a94a31bc685717404e118e
I20260928 21:08:18.551476 1413 log.cc:1196] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/954f1e9e73a94a31bc685717404e118e
I20260928 21:08:18.551901 1414 ts_tablet_manager.cc:1939] T 954f1e9e73a94a31bc685717404e118e P a063582ce58c4c53a46be4c81144e717: Deleting consensus metadata
I20260928 21:08:18.552287 1413 ts_tablet_manager.cc:1939] T 954f1e9e73a94a31bc685717404e118e P 6544972a0def47fd9fcd2fa8e743e12d: Deleting consensus metadata
W20260928 21:08:18.552759 864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:42221 (error 108)
I20260928 21:08:18.553706 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 1101
I20260928 21:08:18.555024 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:08:18.554627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:42221 (local address 127.0.0.1:56682)
0928 21:08:18.554857 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.554934 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:42221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260928 21:08:18.555218 864 catalog_manager.cc:4584] TS f890a1283f9244baa69d25fbb66628c5 (127.21.140.65:42221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 954f1e9e73a94a31bc685717404e118e: Network error: Client connection negotiation failed: client connection to 127.21.140.65:42221: connect: Connection refused (error 111)
I20260928 21:08:18.562069 864 catalog_manager.cc:4841] TS 6544972a0def47fd9fcd2fa8e743e12d (127.21.140.67:36337): tablet 954f1e9e73a94a31bc685717404e118e (table test-table [id=af0e94042c0c4e6f84928b7ecb74a477]) successfully deleted
W20260928 21:08:18.563443 1384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:08:18.562666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45475 (local address 127.0.0.1:49364)
0928 21:08:18.562842 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:08:18.562855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:08:18.562938 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:08:18.563341 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:45475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 21:08:18.564154 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 1239
I20260928 21:08:18.572993 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 848
I20260928 21:08:18.580950 22065 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:08:18.926180 22065 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:08:18.926342 22065 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:08:19.272879 22065 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:08:19.273 UTC [430] LOG: received smart shutdown request
2026-09-28 21:08:19.277 UTC [430] LOG: background worker "logical replication launcher" (PID 439) exited with exit code 1
2026-09-28 21:08:19.278 UTC [434] LOG: shutting down
2026-09-28 21:08:19.463 UTC [430] LOG: database system is shut down
2026-09-28T21:08:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47129 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:08:19 dist-test-slave-d5wc krb5kdc[1424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:08:19 dist-test-slave-d5wc krb5kdc[1424](info): set up 2 sockets
Sep 28 21:08:19 dist-test-slave-d5wc krb5kdc[1424](info): commencing operation
krb5kdc: starting...
W20260928 21:08:20.643771 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:08:20 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:08:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/rangeradmin_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/rangeradmin_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/rangerlookup_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/rangerlookup_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
I20260928 21:08:20.820988 22065 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskczrM8B/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/postgres -l logfile start
2026-09-28 21:08:23.454 UTC [1454] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:08:23.454 UTC [1454] LOG: listening on IPv4 address "127.21.140.84", port 54745
2026-09-28 21:08:23.458 UTC [1454] LOG: listening on Unix socket "/tmp/.s.PGSQL.54745"
2026-09-28 21:08:23.483 UTC [1457] LOG: database system was shut down at 2026-09-28 21:08:21 UTC
2026-09-28 21:08:23.488 UTC [1454] LOG: database system is ready to accept connections
I20260928 21:08:24.451521 22065 mini_postgres.cc:96] Postgres bound to 54745
2026-09-28 21:08:24.458 UTC [1465] FATAL: database "slave" does not exist
127.21.140.84:54745 - accepting connections
I20260928 21:08:24.458894 22065 mini_ranger.cc:161] Starting Ranger...
I20260928 21:08:24.474772 22065 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:08:25.010880 22065 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:08:25.010979 22065 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-admin
2026-09-28 21:08:25,297 [I] DB FLAVOR :POSTGRES
2026-09-28 21:08:25,298 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:08:25,298 [I] Checking connection..
2026-09-28 21:08:25,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:08:25,560 [I] Checking connection passed.
2026-09-28 21:08:25,560 [I] --------- Verifying version history table ---------
2026-09-28 21:08:25,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:08:25,793 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:08:25,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:08:26,030 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:08:26,030 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:08:26,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:08:26.300 UTC [1530] WARNING: there is no transaction in progress
2026-09-28 21:08:26,306 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:08:26,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:08:26,550 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:08:26,551 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:08:26,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:08:26,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:08:27,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-d5wc.c.gcp-upstream.internal','N') ;"
2026-09-28 21:08:27,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:08:27,560 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:08:27,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:08:27,794 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:08:27,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:08:28,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:08:28,281 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:08:28,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:08:29.326 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.344 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.361 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.380 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.400 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.420 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.484 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.492 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.509 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.521 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.532 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.547 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.557 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.564 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.571 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:29.578 UTC [1638] WARNING: there is no transaction in progress
2026-09-28 21:08:30,302 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:08:30,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:08:30,588 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:08:30,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:08:30,880 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:08:30,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:08:31,195 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:08:31,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:08:31,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:08:31,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-d5wc.c.gcp-upstream.internal';"
2026-09-28 21:08:31,954 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:08:31,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:08:32,198 [I] DB_PATCHES have already been applied
I20260928 21:08:32.203822 22065 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-admin:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskczrM8B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:08:32.203918 22065 mini_ranger.cc:193] Using host: 127.21.140.84
I20260928 21:08:32.207538 22065 mini_ranger.cc:227] Ranger admin URL: http://127.21.140.84:46883
Sep 28, 2026 9:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:08:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:08:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:08:33.338855 22065 mini_ranger.cc:160] Time spent starting Ranger: real 8.880s user 0.005s sys 0.005s
Sep 28, 2026 9:08:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:08:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 9:08:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:08:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:08:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:08:55.876029 22065 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.140.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.140.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/rangerkms_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.140.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/rangerkms_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.140.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.140.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.140.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/HTTP_127.21.140.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:08:55 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:08:55.966236 22065 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:08:55.976292 22065 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:08:56.889166 22065 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:08:56.889358 22065 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms
I20260928 21:08:56.889381 22065 mini_ranger_kms.cc:227] Using postgres at 127.21.140.84:54745
2026-09-28 21:08:57,068 [I] DB FLAVOR :POSTGRES
2026-09-28 21:08:57,068 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:08:57,068 [I] Checking connection
2026-09-28 21:08:57,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:08:57,330 [I] connection success
2026-09-28 21:08:57,330 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:08:57,330 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:08:57,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:08:57,597 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:08:57,597 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:08:57,597 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:08:57,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.140.84:54745/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:08:57,871 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:08:58.359915 22065 mini_ranger_kms.cc:310] Created kms service
I20260928 21:08:58.588721 22065 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:08:58.635317 22065 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:08:58 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:08:59.079902 22065 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:08:59.080164 22065 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms:/tmp/dist-test-taskczrM8B/build/debug/bin/postgresql.jar:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskczrM8B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:08:59.080196 22065 mini_ranger_kms.cc:240] Using host: 127.21.140.84
I20260928 21:08:59.084599 22065 mini_ranger_kms.cc:276] Ranger KMS PID: 1820
I20260928 21:08:59.084685 22065 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.140.84:51955
Sep 28, 2026 9:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 21:09:01 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790629741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.140.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:01.378429 22065 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.412s user 0.025s sys 0.136s
I20260928 21:09:01.378752 22065 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:09:01.378931 22065 mini_ranger_kms.cc:401] 127.21.140.84:51955/kms/v1/keys
Sep 28 21:09:01 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.140.84@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
Sep 28 21:09:02 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.140.84@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
Sep 28 21:09:04 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:09:04.342175 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37959
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37959
--ranger_config_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:04.455395 1870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:04.455801 1870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:04.455890 1870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:04.460462 1870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:04.460533 1870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:04.460552 1870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:04.460569 1870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:04.460605 1870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:04.466280 1870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.140.84:51955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37959
--ranger_config_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:04.467334 1870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:04.468815 1870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:04.478936 1876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:04.478991 1878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:04.478895 1870 server_base.cc:1034] running on GCE node
W20260928 21:09:04.478902 1875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:04.479647 1870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:04.480899 1870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:04.482064 1870 hybrid_clock.cc:648] HybridClock initialized: now 1790629744482026 us; error 54 us; skew 500 ppm
Sep 28 21:09:04 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:04.484942 1870 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:04.485838 1870 webserver.cc:460] Webserver started at http://127.21.140.126:46251/ using document root <none> and password file <none>
I20260928 21:09:04.486452 1870 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:04.486541 1870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:04.486764 1870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:09:04 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629744, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:04.541399 1870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "a6e61a6897594277b6c958abe3042179"
format_stamp: "Formatted at 2026-09-28 21:09:04 on dist-test-slave-d5wc"
server_key: "f5950648a8f31d247cf53c38c4cecf55"
server_key_iv: "4eb6459288eb710f30e11908b683ca51"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:04.542156 1870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "a6e61a6897594277b6c958abe3042179"
format_stamp: "Formatted at 2026-09-28 21:09:04 on dist-test-slave-d5wc"
server_key: "f5950648a8f31d247cf53c38c4cecf55"
server_key_iv: "4eb6459288eb710f30e11908b683ca51"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:04.547283 1870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 21:09:04.562191 1885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:04.563617 1870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:04.563779 1870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "a6e61a6897594277b6c958abe3042179"
format_stamp: "Formatted at 2026-09-28 21:09:04 on dist-test-slave-d5wc"
server_key: "f5950648a8f31d247cf53c38c4cecf55"
server_key_iv: "4eb6459288eb710f30e11908b683ca51"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:04.563917 1870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:04.582700 1870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:04.585777 1870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:04.586043 1870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:04.594209 1870 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37959
I20260928 21:09:04.594209 1938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37959 every 8 connection(s)
I20260928 21:09:04.595319 1870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:04.598464 1939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:04.600080 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 1870
I20260928 21:09:04.600258 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:09:04.600535 22065 external_mini_cluster.cc:1428] Setting key dfbf2c6282d9370e56df1612eee4e57f
I20260928 21:09:04.605074 1939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Bootstrap starting.
I20260928 21:09:04.607673 1939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:04.608633 1939 log.cc:824] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:09:04 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:04.610770 1939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: No bootstrap required, opened a new log
I20260928 21:09:04.614290 1939 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:04.614588 1939 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:04.614691 1939 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e61a6897594277b6c958abe3042179, State: Initialized, Role: FOLLOWER
I20260928 21:09:04.615197 1939 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:04.615360 1939 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:04.615446 1939 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:04.615500 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:04.602093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.21.140.126:37959)
0928 21:09:04.602521 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:04.602546 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:04.602581 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:09:04.603241 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:04.603257 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:04.603577 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:04.603857 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:04.604819 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:04.605951 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:04.607201 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:04.607748 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:04.607943 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:04.610755 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:04.610760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:04.610773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:04.610806 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:04.612501 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:04.612963 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:04.612969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:04.612974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:04.613082 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:04.613375 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:04.613378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:04.613380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:04.613766 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:04.614024 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:04.614300 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 21:09:04.614626 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260928 21:09:04.615561 1939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:04.616555 1939 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:04.616899 1939 leader_election.cc:302] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e61a6897594277b6c958abe3042179; no voters:
I20260928 21:09:04.617219 1939 leader_election.cc:288] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:09:04.617379 1944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:04.617662 1944 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 LEADER]: Becoming Leader. State: Replica: a6e61a6897594277b6c958abe3042179, State: Running, Role: LEADER
I20260928 21:09:04.618005 1944 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:04.618562 1939 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:04.619102 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e61a6897594277b6c958abe3042179. Latest consensus state: current_term: 1 leader_uuid: "a6e61a6897594277b6c958abe3042179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } } }
I20260928 21:09:04.619213 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:04.619701 1945 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e61a6897594277b6c958abe3042179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } } }
I20260928 21:09:04.619817 1945 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:04.620972 1949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:04.621062 1939 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:09:04.626358 1949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:04.632364 1949 catalog_manager.cc:1347] Generated new cluster ID: 0438114e6f6d4bbbb988736cf52391cb
I20260928 21:09:04.632500 1949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:04.656137 1949 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:09:04.657158 1949 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:04.664819 1949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Generated new TSK 0
I20260928 21:09:04.665555 1949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:09:05.327121 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37959
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:05.493732 1984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:05.494299 1984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:05.494393 1984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:05.500844 1984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:05.500936 1984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:05.501032 1984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:05.509383 1984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.140.84:51955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:05.511157 1984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:05.512629 1984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:05.527240 1991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:05.527491 1984 server_base.cc:1034] running on GCE node
W20260928 21:09:05.527710 1993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:05.527846 1995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:05.528412 1984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:05.529230 1984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:05.530455 1984 hybrid_clock.cc:648] HybridClock initialized: now 1790629745530428 us; error 47 us; skew 500 ppm
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:05.533473 1984 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:05.534451 1984 webserver.cc:460] Webserver started at http://127.21.140.65:39349/ using document root <none> and password file <none>
I20260928 21:09:05.535017 1984 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:05.535112 1984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:05.535336 1984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:05.552923 1984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "7b6ac64110194d3f9748a5ba606378fa"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "a4250dba4325616a13da5908b9b1d15e"
server_key_iv: "bd44bf168cc3edc180d2dc45d95ef69d"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:05.554819 1984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "7b6ac64110194d3f9748a5ba606378fa"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "a4250dba4325616a13da5908b9b1d15e"
server_key_iv: "bd44bf168cc3edc180d2dc45d95ef69d"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:05.559592 1984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:09:05.598354 2002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:05.600147 1984 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.001s sys 0.000s
I20260928 21:09:05.600283 1984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "7b6ac64110194d3f9748a5ba606378fa"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "a4250dba4325616a13da5908b9b1d15e"
server_key_iv: "bd44bf168cc3edc180d2dc45d95ef69d"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:05.600409 1984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:05.625682 1984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:05.629194 1984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:05.629442 1984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:05.630227 1984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:05.631280 1984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:05.631392 1984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:05.631517 1984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:05.631620 1984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:05.643262 1984 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:33455
I20260928 21:09:05.646108 1984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:05.648692 2117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:33455 every 8 connection(s)
I20260928 21:09:05.652112 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 1984
I20260928 21:09:05.652240 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:09:05.652558 22065 external_mini_cluster.cc:1428] Setting key 8e0f2790690f4b4039f07322939bfb74
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:09:05.668081 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:05.651461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37959 (local address 127.21.140.65:52833)
0928 21:09:05.652114 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:05.652175 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:05.653602 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:05.654556 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:05.654566 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:05.654931 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:05.655662 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:05.655685 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.656898 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.656902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:05.658401 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:05.658412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.659166 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.659170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:05.659412 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:05.660162 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:05.660183 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:05.661920 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:05.664223 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:05.664229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:05.664248 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:05.664600 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:05.665234 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:05.665237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:05.665240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:05.665419 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:05.665904 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:05.665911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:05.666261 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 21:09:05.666475 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 21:09:05.666831 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":155,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:09:05.669373 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:05.652791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:52833 (local address 127.21.140.126:37959)
0928 21:09:05.654146 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:05.654155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:05.654170 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:09:05.654216 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:05.654219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:05.654266 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:05.654359 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:05.655855 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.656734 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.658566 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.658975 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.659787 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:05.662154 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:05.662158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:05.662162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:05.662189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:05.664081 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:05.664751 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:05.664754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:05.664756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:05.665119 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:05.665524 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:05.665527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:05.665529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:05.665757 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:05.665849 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:05.669065 (+ 3216us) server_negotiation.cc:299] Negotiation successful
0928 21:09:05.669208 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":192,"threads_started":1}
I20260928 21:09:05.670746 2118 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37959
I20260928 21:09:05.671015 2118 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:05.671671 2118 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:05.673511 1903 ts_manager.cc:194] Registered new tserver with Master: 7b6ac64110194d3f9748a5ba606378fa (127.21.140.65:33455)
I20260928 21:09:05.675143 1903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:52833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:09:05.693142 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37959
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:09:05.834506 2124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:05.834823 2124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:05.834964 2124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:05.839263 2124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:05.839356 2124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:05.839460 2124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:05.845747 2124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.140.84:51955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:05.846997 2124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:05.848449 2124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:05.858914 2131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:05.859113 2132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:05.859416 2134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:05.860260 2124 server_base.cc:1034] running on GCE node
I20260928 21:09:05.860752 2124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:05.861613 2124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:05.862794 2124 hybrid_clock.cc:648] HybridClock initialized: now 1790629745862725 us; error 87 us; skew 500 ppm
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:05.865808 2124 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:05.866798 2124 webserver.cc:460] Webserver started at http://127.21.140.66:39885/ using document root <none> and password file <none>
I20260928 21:09:05.867561 2124 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:05.867658 2124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:05.867934 2124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:05.886199 2124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "5e3adaa5b38146ce98a17876cbca1a6a"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "2c2740e8459129d438cc174ec3257cef"
server_key_iv: "1ec7887f7a63885416828c20b6557dbb"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:05.887115 2124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "5e3adaa5b38146ce98a17876cbca1a6a"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "2c2740e8459129d438cc174ec3257cef"
server_key_iv: "1ec7887f7a63885416828c20b6557dbb"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:05.895540 2124 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260928 21:09:05.918226 2141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:05.919979 2124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 21:09:05.920195 2124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "5e3adaa5b38146ce98a17876cbca1a6a"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "2c2740e8459129d438cc174ec3257cef"
server_key_iv: "1ec7887f7a63885416828c20b6557dbb"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:05.920432 2124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:05.941803 2124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:05.958961 2124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:05.959190 2124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:05.959954 2124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:05.961090 2124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:05.961160 2124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:05.961218 2124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:05.961253 2124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:05.974983 2124 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:38849
I20260928 21:09:05.976431 2124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:09:05.982096 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2124
I20260928 21:09:05.982187 2254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:38849 every 8 connection(s)
I20260928 21:09:05.982290 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:09:05.982604 22065 external_mini_cluster.cc:1428] Setting key 060d6ac26fbb03fe12e63d64e90f56c5
Sep 28 21:09:05 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:05.999500 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:05.986915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:34575 (local address 127.21.140.126:37959)
0928 21:09:05.987047 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:05.987052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:05.987288 (+ 236us) server_negotiation.cc:407] Connection header received
0928 21:09:05.988274 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:05.988278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:05.988334 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:05.988421 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:05.990146 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.990738 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.991999 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.992329 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.992452 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:05.995121 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:05.995125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:05.995127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:05.995154 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:05.996865 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:05.997476 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:05.997480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:05.997482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:05.997574 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:05.998003 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:05.998010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:05.998013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:05.998281 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:05.998389 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:05.999156 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 21:09:05.999309 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:09:06.001713 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:05.985761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37959 (local address 127.21.140.66:34575)
0928 21:09:05.987130 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:05.987184 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:05.988047 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:05.988767 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:05.988779 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:05.989190 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:05.989926 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:05.989943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.990892 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.990896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:05.991859 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:05.991869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:05.992493 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:05.992498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:05.992704 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:05.993384 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:05.993402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:05.994909 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:05.997020 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:05.997028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:05.997048 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:05.997351 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:05.997678 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:05.997682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:05.997685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:05.997864 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:05.998454 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:05.998461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:05.998846 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 21:09:06.000052 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0928 21:09:06.000439 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":179,"threads_started":1}
I20260928 21:09:06.003263 2255 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37959
I20260928 21:09:06.003539 2255 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:09:06.004180 2255 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:06.005496 1903 ts_manager.cc:194] Registered new tserver with Master: 5e3adaa5b38146ce98a17876cbca1a6a (127.21.140.66:38849)
I20260928 21:09:06.006525 1903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:34575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:09:06.030849 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37959
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:06.205428 2261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:06.206323 2261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:06.206447 2261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:06.213178 2261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:06.213342 2261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:06.213480 2261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:06.222105 2261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.140.84:51955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:06.223773 2261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:06.225163 2261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:06.239038 2267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:06.239552 2266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:06.240118 2269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:06.241200 2261 server_base.cc:1034] running on GCE node
I20260928 21:09:06.241688 2261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:06.242533 2261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:06.258095 2261 hybrid_clock.cc:648] HybridClock initialized: now 1790629746257980 us; error 131 us; skew 500 ppm
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:06.266325 2261 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:06.267544 2261 webserver.cc:460] Webserver started at http://127.21.140.67:32989/ using document root <none> and password file <none>
I20260928 21:09:06.268326 2261 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:06.268560 2261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:06.268849 2261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629745, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:06.293737 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "6be42a2eec2347509eac02993b792054"
format_stamp: "Formatted at 2026-09-28 21:09:06 on dist-test-slave-d5wc"
server_key: "62e342fb20af6489c314ad2613deb9c0"
server_key_iv: "270b9d6bd67e39962ae4a7e18f96a2b9"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:06.294710 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "6be42a2eec2347509eac02993b792054"
format_stamp: "Formatted at 2026-09-28 21:09:06 on dist-test-slave-d5wc"
server_key: "62e342fb20af6489c314ad2613deb9c0"
server_key_iv: "270b9d6bd67e39962ae4a7e18f96a2b9"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:06.312021 2261 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.005s sys 0.000s
I20260928 21:09:06.333365 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.335234 2261 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260928 21:09:06.335383 2261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "6be42a2eec2347509eac02993b792054"
format_stamp: "Formatted at 2026-09-28 21:09:06 on dist-test-slave-d5wc"
server_key: "62e342fb20af6489c314ad2613deb9c0"
server_key_iv: "270b9d6bd67e39962ae4a7e18f96a2b9"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:06.335529 2261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:06.369333 2261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:06.373073 2261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:06.373394 2261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:06.374177 2261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:06.375310 2261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:06.375424 2261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.375536 2261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:06.375607 2261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.389982 2261 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:40163
I20260928 21:09:06.391291 2261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:09:06.392467 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:40163 every 8 connection(s)
I20260928 21:09:06.397583 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2261
I20260928 21:09:06.397732 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:09:06.398113 22065 external_mini_cluster.cc:1428] Setting key 48c968d10a854ea3e93e870c39f493ea
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:06.411531 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:06.396263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50137 (local address 127.21.140.126:37959)
0928 21:09:06.396374 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:06.396378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:06.396394 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:09:06.397180 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:06.397185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:06.397245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:06.397308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:06.399274 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.400066 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.401902 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.402279 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.402368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:06.406814 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:06.406818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:06.406822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:06.406856 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:06.408804 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:06.409599 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:06.409603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:06.409604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:06.409701 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:06.410185 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:06.410189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:06.410190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:06.410427 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:06.410497 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:06.411229 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 21:09:06.411366 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:09:06.413444 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:06.395193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37959 (local address 127.21.140.67:50137)
0928 21:09:06.395777 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:06.395844 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:06.396891 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:06.397902 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:06.397913 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:06.398402 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:06.399086 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:06.399106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.400259 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.400262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:06.401740 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:06.401753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.402602 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.402606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:06.402829 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:06.403552 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:06.403575 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:06.406542 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:06.409001 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:06.409009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:06.409029 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:06.409464 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:06.409811 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:06.409814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:06.409816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:06.410046 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:06.410579 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:06.410586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:06.410930 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 21:09:06.412016 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0928 21:09:06.412386 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":128,"threads_started":1}
I20260928 21:09:06.415056 2390 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37959
I20260928 21:09:06.415316 2390 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:06.415917 2390 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:06.417104 1903 ts_manager.cc:194] Registered new tserver with Master: 6be42a2eec2347509eac02993b792054 (127.21.140.67:40163)
I20260928 21:09:06.417853 1903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:50137
I20260928 21:09:06.421439 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:09:06.422514 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 1984
I20260928 21:09:06.431699 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2124
I20260928 21:09:06.438668 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2261
I20260928 21:09:06.447300 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 1870
I20260928 21:09:06.453189 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37959
--webserver_interface=127.21.140.126
--webserver_port=46251
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37959
--ranger_config_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:06.565157 2393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:06.565399 2393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:06.565438 2393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:06.569532 2393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:06.569586 2393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:06.569605 2393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:06.569624 2393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:06.569640 2393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:06.574939 2393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37959
--ranger_config_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:06.575935 2393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:06.576911 2393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:06.585675 2401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:06.585707 2398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:06.585842 2399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:06.585914 2393 server_base.cc:1034] running on GCE node
I20260928 21:09:06.586525 2393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:06.587668 2393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:06.588837 2393 hybrid_clock.cc:648] HybridClock initialized: now 1790629746588809 us; error 43 us; skew 500 ppm
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:06.591619 2393 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:06.592505 2393 webserver.cc:460] Webserver started at http://127.21.140.126:46251/ using document root <none> and password file <none>
I20260928 21:09:06.593026 2393 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:06.593092 2393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:06.656534 2393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:09:06.659310 2409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.660966 2393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:09:06.661099 2393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "a6e61a6897594277b6c958abe3042179"
format_stamp: "Formatted at 2026-09-28 21:09:04 on dist-test-slave-d5wc"
server_key: "f5950648a8f31d247cf53c38c4cecf55"
server_key_iv: "4eb6459288eb710f30e11908b683ca51"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:06.661815 2393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:06.683537 2393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:06.686517 2393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:06.686673 2393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:06.694072 2393 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37959
I20260928 21:09:06.694108 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37959 every 8 connection(s)
I20260928 21:09:06.695132 2393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:06.699256 2462 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:09:06.700881 2462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Bootstrap starting.
I20260928 21:09:06.701524 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2393
I20260928 21:09:06.702088 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:33455
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:37959
--webserver_port=39349
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:06.718375 2462 log.cc:824] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:06.724949 2462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:06.725383 2462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Bootstrap complete.
I20260928 21:09:06.729561 2462 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:06.729955 2462 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e61a6897594277b6c958abe3042179, State: Initialized, Role: FOLLOWER
I20260928 21:09:06.730546 2462 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:06.730727 2462 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:06.730796 2462 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:06.730911 2462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:09:06.732872 2462 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:06.733228 2462 leader_election.cc:302] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e61a6897594277b6c958abe3042179; no voters:
I20260928 21:09:06.733585 2462 leader_election.cc:288] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:09:06.733685 2466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:09:06.733939 2466 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [term 2 LEADER]: Becoming Leader. State: Replica: a6e61a6897594277b6c958abe3042179, State: Running, Role: LEADER
I20260928 21:09:06.734303 2466 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } }
I20260928 21:09:06.734848 2462 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:06.735224 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e61a6897594277b6c958abe3042179. Latest consensus state: current_term: 2 leader_uuid: "a6e61a6897594277b6c958abe3042179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } } }
I20260928 21:09:06.735342 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:06.735719 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6e61a6897594277b6c958abe3042179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e61a6897594277b6c958abe3042179" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37959 } } }
I20260928 21:09:06.735878 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:06.736179 2470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:06.739471 2470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:06.739980 2462 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:09:06.740163 2470 catalog_manager.cc:1259] Loaded cluster ID: 0438114e6f6d4bbbb988736cf52391cb
I20260928 21:09:06.740280 2470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:06.745201 2470 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:06.746243 2470 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Loaded TSK: 0
I20260928 21:09:06.747556 2470 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:09:06.824222 2464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:06.824488 2464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:06.824594 2464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:06.829262 2464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:06.829350 2464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:06.829452 2464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:06.835424 2464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:33455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=39349
--tserver_master_addrs=127.21.140.126:37959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:06.836726 2464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:06.837872 2464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:06.850286 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:06.850436 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:06.850879 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:06.851274 2464 server_base.cc:1034] running on GCE node
I20260928 21:09:06.851750 2464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:06.852540 2464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:06.853736 2464 hybrid_clock.cc:648] HybridClock initialized: now 1790629746853699 us; error 53 us; skew 500 ppm
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:06.857342 2464 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:06.858629 2464 webserver.cc:460] Webserver started at http://127.21.140.65:39349/ using document root <none> and password file <none>
I20260928 21:09:06.859375 2464 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:06.859488 2464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:06.885305 2464 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:09:06.887914 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.889487 2464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:09:06.889628 2464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "7b6ac64110194d3f9748a5ba606378fa"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "a4250dba4325616a13da5908b9b1d15e"
server_key_iv: "bd44bf168cc3edc180d2dc45d95ef69d"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:06.890271 2464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:06.911856 2464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:06.915131 2464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:06.915344 2464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:06.915980 2464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:06.916890 2464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:06.916958 2464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.917021 2464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:06.917047 2464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:06.929414 2464 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:33455
I20260928 21:09:06.930750 2464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:06.932159 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:33455 every 8 connection(s)
I20260928 21:09:06.939456 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2464
I20260928 21:09:06.940006 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:38849
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:37959
--webserver_port=39885
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:09:06 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629746, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:06.963071 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:06.934382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37959 (local address 127.21.140.65:40091)
0928 21:09:06.936929 (+ 2547us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:06.936982 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:06.938049 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:06.940262 (+ 2213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:06.940271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:06.940563 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:06.941026 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:06.941038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.949595 (+ 8557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.949601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:06.951425 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:06.951447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.952731 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.952735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:06.952927 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:06.953672 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:06.953696 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:06.955491 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:06.959232 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:06.959239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:06.959254 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:06.959627 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:06.960056 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:06.960059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:06.960061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:06.960238 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:06.961067 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:06.961074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:06.961351 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 21:09:06.961525 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 21:09:06.961894 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2299,"thread_start_us":147,"threads_started":1}
I20260928 21:09:06.967589 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:06.935862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:40091 (local address 127.21.140.126:37959)
0928 21:09:06.938549 (+ 2687us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:06.938570 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:06.938615 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:09:06.938898 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:06.938913 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:06.939324 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:06.939679 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:06.941204 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.949326 (+ 8122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.951673 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:06.952517 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:06.956587 (+ 4070us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:06.956816 (+ 229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:06.956822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:06.956843 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:06.956869 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:06.959063 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:06.959782 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:06.959789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:06.959797 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:06.959936 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:06.960365 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:06.960368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:06.960370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:06.960931 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:06.966115 (+ 5184us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:06.966300 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 21:09:06.966641 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2501,"thread_start_us":107,"threads_started":1}
I20260928 21:09:06.970292 2624 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37959
I20260928 21:09:06.970558 2624 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:06.971145 2624 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
W20260928 21:09:06.973426 2624 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:37959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:09:06.973554 2624 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:06.973727 2624 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:06.974612 2624 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:06.974803 2624 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
W20260928 21:09:06.975582 2624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:09:07.089586 2628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:07.089865 2628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:07.089948 2628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:07.094947 2628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:07.095077 2628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:07.095206 2628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:07.102176 2628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:38849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=39885
--tserver_master_addrs=127.21.140.126:37959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:07.103370 2628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:07.104504 2628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:07.115599 2634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:07.115700 2628 server_base.cc:1034] running on GCE node
W20260928 21:09:07.115921 2636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:07.116026 2633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:07.116467 2628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:07.117236 2628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:07.118425 2628 hybrid_clock.cc:648] HybridClock initialized: now 1790629747118387 us; error 57 us; skew 500 ppm
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:07.122056 2628 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:07.123136 2628 webserver.cc:460] Webserver started at http://127.21.140.66:39885/ using document root <none> and password file <none>
I20260928 21:09:07.123871 2628 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:07.123984 2628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:07.146615 2628 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:07.149394 2644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:07.150748 2628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:09:07.150846 2628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "5e3adaa5b38146ce98a17876cbca1a6a"
format_stamp: "Formatted at 2026-09-28 21:09:05 on dist-test-slave-d5wc"
server_key: "2c2740e8459129d438cc174ec3257cef"
server_key_iv: "1ec7887f7a63885416828c20b6557dbb"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:07.151417 2628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:07.168471 2628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:07.172240 2628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:07.172441 2628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:07.173154 2628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:07.174268 2628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:07.174335 2628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:07.174391 2628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:07.174420 2628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:07.185847 2628 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:38849
I20260928 21:09:07.185902 2759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:38849 every 8 connection(s)
I20260928 21:09:07.187129 2628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:09:07.192392 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2628
I20260928 21:09:07.192899 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.140.84:51955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:40163
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:37959
--webserver_port=32989
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:07.216279 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:07.189551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:40983 (local address 127.21.140.126:37959)
0928 21:09:07.189682 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:07.189686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:07.199698 (+ 10012us) server_negotiation.cc:407] Connection header received
0928 21:09:07.201045 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:07.201049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:07.201114 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:07.201174 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:07.202901 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.203769 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.205539 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.205929 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.206092 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:07.209206 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:07.209211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:07.209214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:07.209244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:07.211112 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:07.214243 (+ 3131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:07.214247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:07.214249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:07.214355 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:07.214846 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:07.214850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:07.214853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:07.215101 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:07.215196 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:07.215947 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 21:09:07.216101 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:09:07.218276 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:07.190038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37959 (local address 127.21.140.66:40983)
0928 21:09:07.199494 (+ 9456us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:07.199565 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:07.200773 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:07.201615 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:07.201625 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:07.202027 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:07.202721 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:07.202740 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.203927 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.203930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:07.205387 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:07.205399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.206270 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.206274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:07.206471 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:07.207162 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:07.207184 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:07.208980 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:07.211263 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:07.211270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:07.211287 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:07.211670 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:07.214495 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:07.214498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:07.214501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:07.214707 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:07.215272 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:07.215278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:07.215632 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 21:09:07.216790 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0928 21:09:07.217149 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9191,"thread_start_us":153,"threads_started":1}
I20260928 21:09:07.220391 2760 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37959
I20260928 21:09:07.220669 2760 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.221244 2760 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
W20260928 21:09:07.223300 2760 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:37959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:09:07.223421 2760 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.223579 2760 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:07.224467 2760 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.224641 2760 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
W20260928 21:09:07.225407 2760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:09:07.347087 2763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:07.347384 2763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:07.347455 2763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:07.352681 2763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:07.352790 2763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:07.352895 2763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:07.361155 2763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.140.84:51955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:40163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=32989
--tserver_master_addrs=127.21.140.126:37959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:07.362442 2763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:07.363523 2763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:07.374766 2769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:07.374909 2763 server_base.cc:1034] running on GCE node
W20260928 21:09:07.374766 2770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:07.374781 2772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:07.375598 2763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:07.376552 2763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:07.383961 2763 hybrid_clock.cc:648] HybridClock initialized: now 1790629747383726 us; error 235 us; skew 500 ppm
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:07.388444 2763 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:07.389720 2763 webserver.cc:460] Webserver started at http://127.21.140.67:32989/ using document root <none> and password file <none>
I20260928 21:09:07.390555 2763 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:07.390671 2763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:07.418778 2763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:09:07.421892 2780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:07.423324 2763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:09:07.423480 2763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "6be42a2eec2347509eac02993b792054"
format_stamp: "Formatted at 2026-09-28 21:09:06 on dist-test-slave-d5wc"
server_key: "62e342fb20af6489c314ad2613deb9c0"
server_key_iv: "270b9d6bd67e39962ae4a7e18f96a2b9"
server_key_version: "kuduclusterkey@0"
I20260928 21:09:07.424088 2763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:07.450183 2763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:07.453653 2763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:07.453871 2763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:07.454644 2763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:07.455785 2763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:07.455850 2763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:07.455919 2763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:07.455951 2763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:07.467540 2763 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:40163
I20260928 21:09:07.467604 2894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:40163 every 8 connection(s)
I20260928 21:09:07.468570 2763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:09:07.477161 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2763
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:07.484287 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:07.470866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:56323 (local address 127.21.140.126:37959)
0928 21:09:07.470994 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:07.470999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:07.471989 (+ 990us) server_negotiation.cc:407] Connection header received
0928 21:09:07.472985 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:07.472989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:07.473047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:07.473155 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:07.474310 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.474847 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.476164 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.476399 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.476516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:07.479728 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:07.479734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:07.479740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:07.479783 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:07.481648 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:07.482375 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:07.482379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:07.482380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:07.482490 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:07.482938 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:07.482942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:07.482944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:07.483173 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:07.483278 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:07.483986 (+ 708us) server_negotiation.cc:299] Negotiation successful
0928 21:09:07.484117 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:09:07.485347 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:07.471373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37959 (local address 127.21.140.67:56323)
0928 21:09:07.471836 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:07.471888 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:07.472771 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:07.473336 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:07.473342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:07.473603 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:07.474159 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:07.474174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.474973 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.474976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:07.476033 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:07.476042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.476553 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.476557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:07.476757 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:07.477450 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:07.477471 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:07.479554 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:07.481784 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:07.481792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:07.481809 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:07.482236 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:07.482604 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:07.482607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:07.482609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:07.482807 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:07.483306 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:07.483314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:07.483664 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 21:09:07.483890 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 21:09:07.484253 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
I20260928 21:09:07.486764 2895 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37959
I20260928 21:09:07.487036 2895 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.487645 2895 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
W20260928 21:09:07.490738 2895 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:37959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:09:07.490888 2895 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.491072 2895 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:07.491992 2895 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.492162 2895 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
W20260928 21:09:07.492933 2895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:09:07.504513 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:07.477922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.21.140.126:37959)
0928 21:09:07.478653 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:07.478656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:07.478915 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:09:07.479066 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:07.479070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:07.479122 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:07.479219 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:07.480927 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.482614 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.495339 (+ 12725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:07.495729 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:07.495856 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:07.497600 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:07.497606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:07.497609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:07.497637 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:07.502640 (+ 5003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:07.503174 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:07.503184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:07.503187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:07.503272 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:07.503572 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:07.503576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:07.503578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:07.503781 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:07.504013 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:07.504151 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:09:07.504294 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":126,"threads_started":1}
Sep 28 21:09:07 dist-test-slave-d5wc krb5kdc[1424](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:07.975811 2624 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:07.976119 2624 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:07.977319 2422 ts_manager.cc:194] Registered new tserver with Master: 7b6ac64110194d3f9748a5ba606378fa (127.21.140.65:33455)
I20260928 21:09:07.979034 2422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:40091
I20260928 21:09:08.225662 2760 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:08.226061 2760 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:08.226904 2422 ts_manager.cc:194] Registered new tserver with Master: 5e3adaa5b38146ce98a17876cbca1a6a (127.21.140.66:38849)
I20260928 21:09:08.227703 2422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:40983
Sep 28 21:09:08 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:08.493173 2895 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:08.493468 2895 heartbeater.cc:511] Master 127.21.140.126:37959 requested a full tablet report, sending...
I20260928 21:09:08.494226 2422 ts_manager.cc:194] Registered new tserver with Master: 6be42a2eec2347509eac02993b792054 (127.21.140.67:40163)
I20260928 21:09:08.495002 2422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:56323
I20260928 21:09:08.505632 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:09:08.518419 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.507450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.21.140.126:37959)
0928 21:09:08.508046 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.508051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.508410 (+ 359us) server_negotiation.cc:407] Connection header received
0928 21:09:08.508515 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.508519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.508575 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.508666 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:08.509672 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.510564 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.511780 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.512152 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.512263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.514196 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:08.514200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:08.514203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:08.514232 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:08.516111 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:08.516628 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:08.516632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:08.516634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:08.516721 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:08.517009 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:08.517012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:08.517014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:08.517247 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:08.517377 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.518110 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.518243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":77,"threads_started":1}
I20260928 21:09:08.523116 2422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:09:08.528127 2422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:09:08.550117 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.541389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:40163 (local address 127.0.0.1:60932)
0928 21:09:08.544523 (+ 3134us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.544575 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:08.544705 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:08.545457 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:08.545463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:08.545484 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:08.545612 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.545618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.547028 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.547032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.548338 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.548349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.549549 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.549553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.549655 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.549690 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:09:08.549806 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 21:09:08.549925 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2956,"thread_start_us":85,"threads_started":1}
I20260928 21:09:08.553182 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.542176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.21.140.67:40163)
0928 21:09:08.544576 (+ 2400us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.544584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.544849 (+ 265us) server_negotiation.cc:407] Connection header received
0928 21:09:08.544929 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.544935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.545145 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.545280 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:08.545749 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.546891 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.548478 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.549251 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.549350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.549492 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.552910 (+ 3418us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.553032 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2286,"thread_start_us":58,"threads_started":1}
I20260928 21:09:08.556425 2820 tablet_service.cc:1467] Processing CreateTablet for tablet a139ffda0af349da874dfbb18774b133 (DEFAULT_TABLE table=test-table [id=00002f4da1604bdbaad10ac3d79afb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:08.557305 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.543941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:38849 (local address 127.0.0.1:50100)
0928 21:09:08.550603 (+ 6662us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.550616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:08.550687 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:08.551410 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:08.551413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:08.551429 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:08.551502 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.551507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.554229 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.554233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.555415 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.555426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.556839 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.556843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.556927 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.556953 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:09:08.557088 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 21:09:08.557164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6608,"thread_start_us":94,"threads_started":1}
I20260928 21:09:08.557669 2820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a139ffda0af349da874dfbb18774b133. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:08.558065 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.544096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.21.140.66:38849)
0928 21:09:08.545212 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.545219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.550797 (+ 5578us) server_negotiation.cc:407] Connection header received
0928 21:09:08.550882 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.550888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.551101 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.551248 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:08.551624 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.552746 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.555558 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.556434 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.556540 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.556679 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.557792 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.557896 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":69,"threads_started":1}
I20260928 21:09:08.559412 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.542862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:33455 (local address 127.0.0.1:36216)
0928 21:09:08.544850 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.544865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:08.544961 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:08.553991 (+ 9030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:08.553994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:08.554013 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:08.554103 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.554109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.555634 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.555637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.556776 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.556787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.559003 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.559006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.559103 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.559127 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:09:08.559212 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:09:08.559287 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"thread_start_us":99,"threads_started":1}
I20260928 21:09:08.560672 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.543252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.21.140.65:33455)
0928 21:09:08.545379 (+ 2127us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.545384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.545406 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:09:08.553389 (+ 7983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.553395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.553637 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.553804 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:08.554338 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.555481 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.556922 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.558598 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.558776 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.558935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.560441 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.560532 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":103,"threads_started":1}
I20260928 21:09:08.561471 2693 tablet_service.cc:1467] Processing CreateTablet for tablet a139ffda0af349da874dfbb18774b133 (DEFAULT_TABLE table=test-table [id=00002f4da1604bdbaad10ac3d79afb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:08.562682 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a139ffda0af349da874dfbb18774b133. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:08.563570 2558 tablet_service.cc:1467] Processing CreateTablet for tablet a139ffda0af349da874dfbb18774b133 (DEFAULT_TABLE table=test-table [id=00002f4da1604bdbaad10ac3d79afb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:08.564684 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a139ffda0af349da874dfbb18774b133. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:08.580310 2930 tablet_bootstrap.cc:492] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Bootstrap starting.
I20260928 21:09:08.582814 2930 tablet_bootstrap.cc:654] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:08.583680 2930 log.cc:824] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:08.584105 2931 tablet_bootstrap.cc:492] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Bootstrap starting.
I20260928 21:09:08.586445 2931 tablet_bootstrap.cc:654] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:08.587230 2931 log.cc:824] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:08.587963 2929 tablet_bootstrap.cc:492] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: Bootstrap starting.
I20260928 21:09:08.591333 2929 tablet_bootstrap.cc:654] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:08.592180 2929 log.cc:824] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:08.595252 2930 tablet_bootstrap.cc:492] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: No bootstrap required, opened a new log
I20260928 21:09:08.595486 2930 ts_tablet_manager.cc:1397] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
Sep 28 21:09:08 dist-test-slave-d5wc krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629747, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for HTTP/127.21.140.84@KRBTEST.COM
I20260928 21:09:08.599555 2931 tablet_bootstrap.cc:492] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: No bootstrap required, opened a new log
I20260928 21:09:08.599771 2931 ts_tablet_manager.cc:1397] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260928 21:09:08.604820 2931 raft_consensus.cc:348] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.605142 2931 raft_consensus.cc:374] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:08.607151 2930 raft_consensus.cc:348] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.607440 2930 raft_consensus.cc:374] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:08.607484 2930 raft_consensus.cc:729] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be42a2eec2347509eac02993b792054, State: Initialized, Role: FOLLOWER
I20260928 21:09:08.607740 2931 raft_consensus.cc:729] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3adaa5b38146ce98a17876cbca1a6a, State: Initialized, Role: FOLLOWER
I20260928 21:09:08.608289 2931 consensus_queue.cc:260] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.609287 2930 consensus_queue.cc:260] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.609433 2931 ts_tablet_manager.cc:1428] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260928 21:09:08.609795 2760 heartbeater.cc:503] Master 127.21.140.126:37959 was elected leader, sending a full tablet report...
I20260928 21:09:08.611111 2930 ts_tablet_manager.cc:1428] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20260928 21:09:08.611806 2895 heartbeater.cc:503] Master 127.21.140.126:37959 was elected leader, sending a full tablet report...
I20260928 21:09:08.616420 2929 tablet_bootstrap.cc:492] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: No bootstrap required, opened a new log
I20260928 21:09:08.616644 2929 ts_tablet_manager.cc:1397] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20260928 21:09:08.620824 2935 raft_consensus.cc:482] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:08.621218 2935 raft_consensus.cc:504] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.622045 2929 raft_consensus.cc:348] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.622349 2929 raft_consensus.cc:374] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:08.622403 2929 raft_consensus.cc:729] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6ac64110194d3f9748a5ba606378fa, State: Initialized, Role: FOLLOWER
I20260928 21:09:08.622913 2935 leader_election.cc:288] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6be42a2eec2347509eac02993b792054 (127.21.140.67:40163), 7b6ac64110194d3f9748a5ba606378fa (127.21.140.65:33455)
I20260928 21:09:08.623919 2929 consensus_queue.cc:260] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.625010 2929 ts_tablet_manager.cc:1428] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 21:09:08.625356 2624 heartbeater.cc:503] Master 127.21.140.126:37959 was elected leader, sending a full tablet report...
I20260928 21:09:08.628471 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.623449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:57783 (local address 127.21.140.67:40163)
0928 21:09:08.623577 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.623581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.623977 (+ 396us) server_negotiation.cc:407] Connection header received
0928 21:09:08.624280 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.624284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.624338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.624391 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:08.624857 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.625539 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.627178 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.627749 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.627844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.627894 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.628270 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.628347 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:09:08.629029 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.623070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:40163 (local address 127.21.140.66:57783)
0928 21:09:08.623898 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.623912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:08.624142 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:08.624538 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:08.624542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:08.624566 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:08.624728 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.624736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.625750 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.625753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.627032 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.627046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.628002 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.628007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.628117 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.628143 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:09:08.628785 (+ 642us) client_negotiation.cc:240] Negotiation successful
0928 21:09:08.628864 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":118,"threads_started":1}
I20260928 21:09:08.630434 2833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a139ffda0af349da874dfbb18774b133" candidate_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be42a2eec2347509eac02993b792054" is_pre_election: true
I20260928 21:09:08.630730 2833 raft_consensus.cc:2454] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3adaa5b38146ce98a17876cbca1a6a in term 0.
I20260928 21:09:08.631239 2648 leader_election.cc:302] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3adaa5b38146ce98a17876cbca1a6a, 6be42a2eec2347509eac02993b792054; no voters:
I20260928 21:09:08.631534 2935 raft_consensus.cc:2781] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:09:08.631616 2935 raft_consensus.cc:482] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:09:08.631800 2935 raft_consensus.cc:3037] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:08.633083 2935 raft_consensus.cc:504] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.633437 2935 leader_election.cc:288] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [CANDIDATE]: Term 1 election: Requested vote from peers 6be42a2eec2347509eac02993b792054 (127.21.140.67:40163), 7b6ac64110194d3f9748a5ba606378fa (127.21.140.65:33455)
I20260928 21:09:08.634262 2833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a139ffda0af349da874dfbb18774b133" candidate_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6be42a2eec2347509eac02993b792054"
I20260928 21:09:08.634388 2833 raft_consensus.cc:3037] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:08.635744 2833 raft_consensus.cc:2454] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3adaa5b38146ce98a17876cbca1a6a in term 1.
I20260928 21:09:08.636065 2648 leader_election.cc:302] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3adaa5b38146ce98a17876cbca1a6a, 6be42a2eec2347509eac02993b792054; no voters:
I20260928 21:09:08.636245 2935 raft_consensus.cc:2781] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:08.636449 2935 raft_consensus.cc:686] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 1 LEADER]: Becoming Leader. State: Replica: 5e3adaa5b38146ce98a17876cbca1a6a, State: Running, Role: LEADER
I20260928 21:09:08.636808 2935 consensus_queue.cc:237] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } }
I20260928 21:09:08.640465 2422 catalog_manager.cc:5495] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e3adaa5b38146ce98a17876cbca1a6a (127.21.140.66). New cstate: current_term: 1 leader_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 38849 } health_report { overall_health: HEALTHY } } }
I20260928 21:09:08.643409 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.623707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:60759 (local address 127.21.140.65:33455)
0928 21:09:08.623811 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.623815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.627827 (+ 4012us) server_negotiation.cc:407] Connection header received
0928 21:09:08.627988 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.627991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.628032 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.628075 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:08.639911 (+ 11836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.640720 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.642205 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.642751 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.642839 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.642888 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.643218 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.643294 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:09:08.643807 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.623573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:33455 (local address 127.21.140.66:60759)
0928 21:09:08.627743 (+ 4170us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.627758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:08.627886 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:08.628207 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:08.628210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:08.628223 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:08.628333 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.628339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.640873 (+ 12534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.640877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.642055 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:08.642066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.642978 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.642981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:08.643080 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.643113 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 21:09:08.643588 (+ 475us) client_negotiation.cc:240] Negotiation successful
0928 21:09:08.643669 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4101,"thread_start_us":87,"threads_started":1}
I20260928 21:09:08.644541 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a139ffda0af349da874dfbb18774b133" candidate_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ac64110194d3f9748a5ba606378fa" is_pre_election: true
I20260928 21:09:08.644835 2559 raft_consensus.cc:2454] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3adaa5b38146ce98a17876cbca1a6a in term 0.
I20260928 21:09:08.645131 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a139ffda0af349da874dfbb18774b133" candidate_uuid: "5e3adaa5b38146ce98a17876cbca1a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ac64110194d3f9748a5ba606378fa"
I20260928 21:09:08.645234 2560 raft_consensus.cc:3037] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:08.646920 2560 raft_consensus.cc:2454] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3adaa5b38146ce98a17876cbca1a6a in term 1.
I20260928 21:09:08.688690 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.683052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.21.140.66:38849)
0928 21:09:08.683186 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:08.683191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:08.683206 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:09:08.683306 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:08.683309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:08.683376 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:08.683497 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:08.683941 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.684842 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.686245 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:08.686581 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:08.686699 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:08.687186 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:08.687811 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:09:08.687942 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:08.688440 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 21:09:08.688509 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 21:09:08.694892 2625 tablet.cc:2369] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:09:08.695531 2761 tablet.cc:2369] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:09:08.701705 2833 raft_consensus.cc:1261] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 1 FOLLOWER]: Refusing update from remote peer 5e3adaa5b38146ce98a17876cbca1a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:08.702610 2560 raft_consensus.cc:1261] T a139ffda0af349da874dfbb18774b133 P 7b6ac64110194d3f9748a5ba606378fa [term 1 FOLLOWER]: Refusing update from remote peer 5e3adaa5b38146ce98a17876cbca1a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:08.702553 2940 consensus_queue.cc:1035] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be42a2eec2347509eac02993b792054" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:08.703424 2940 consensus_queue.cc:1035] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6ac64110194d3f9748a5ba606378fa" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:09:08.720338 2896 tablet.cc:2369] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:09:08.725653 2945 mvcc.cc:205] Tried to move back new op lower bound from 7334419450670645248 to 7334419450420834304. Current Snapshot: MvccSnapshot[applied={T|T < 7334419450670645248}]
I20260928 21:09:08.725857 2944 mvcc.cc:205] Tried to move back new op lower bound from 7334419450670645248 to 7334419450420834304. Current Snapshot: MvccSnapshot[applied={T|T < 7334419450670645248}]
I20260928 21:09:08.738219 2422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:08.738409 2422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:08.744541 2947 mvcc.cc:205] Tried to move back new op lower bound from 7334419450670645248 to 7334419450420834304. Current Snapshot: MvccSnapshot[applied={T|T < 7334419450670645248}]
I20260928 21:09:08.747323 2693 tablet_service.cc:1514] Processing DeleteTablet for tablet a139ffda0af349da874dfbb18774b133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:08 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:50100
I20260928 21:09:08.748312 2422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6e61a6897594277b6c958abe3042179: Sending DeleteTablet for 3 replicas of tablet a139ffda0af349da874dfbb18774b133
I20260928 21:09:08.748406 2820 tablet_service.cc:1514] Processing DeleteTablet for tablet a139ffda0af349da874dfbb18774b133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:08 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:60932
I20260928 21:09:08.748816 2815 tablet_service.cc:1514] Processing DeleteTablet for tablet a139ffda0af349da874dfbb18774b133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:08 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:60932
I20260928 21:09:08.749266 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet a139ffda0af349da874dfbb18774b133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:08 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:36216
I20260928 21:09:08.750042 2693 tablet_service.cc:1514] Processing DeleteTablet for tablet a139ffda0af349da874dfbb18774b133 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:08 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:50100
I20260928 21:09:08.751294 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2464
I20260928 21:09:08.763003 2953 tablet_replica.cc:330] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: stopping tablet replica
I20260928 21:09:08.763393 2953 raft_consensus.cc:2229] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:09:08.763820 2953 raft_consensus.cc:2258] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:08.766203 2954 tablet_replica.cc:330] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: stopping tablet replica
I20260928 21:09:08.766553 2954 raft_consensus.cc:2229] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:08.766913 2954 raft_consensus.cc:2258] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:08.768864 2953 ts_tablet_manager.cc:1905] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:08.771746 2953 ts_tablet_manager.cc:1918] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:08.771852 2953 log.cc:1196] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/a139ffda0af349da874dfbb18774b133
I20260928 21:09:08.772185 2953 ts_tablet_manager.cc:1939] T a139ffda0af349da874dfbb18774b133 P 5e3adaa5b38146ce98a17876cbca1a6a: Deleting consensus metadata
W20260928 21:09:08.774472 2410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:33455 (error 108)
I20260928 21:09:08.779289 2954 ts_tablet_manager.cc:1905] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:08.782023 2954 ts_tablet_manager.cc:1918] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:08.782135 2954 log.cc:1196] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/a139ffda0af349da874dfbb18774b133
I20260928 21:09:08.782157 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2628
I20260928 21:09:08.784106 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:08.783698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:33455 (local address 127.0.0.1:36220)
0928 21:09:08.783903 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.783984 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 21:09:08.784328 2410 catalog_manager.cc:4584] TS 7b6ac64110194d3f9748a5ba606378fa (127.21.140.65:33455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a139ffda0af349da874dfbb18774b133: Network error: Client connection negotiation failed: client connection to 127.21.140.65:33455: connect: Connection refused (error 111)
I20260928 21:09:08.785126 2954 ts_tablet_manager.cc:1939] T a139ffda0af349da874dfbb18774b133 P 6be42a2eec2347509eac02993b792054: Deleting consensus metadata
W20260928 21:09:08.789773 2413 catalog_manager.cc:4808] TS 6be42a2eec2347509eac02993b792054 (127.21.140.67:40163): delete failed for tablet a139ffda0af349da874dfbb18774b133 because the tablet was not found. No further retry: Not found: Tablet not found: a139ffda0af349da874dfbb18774b133
W20260928 21:09:08.791177 2924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:09:08.790625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:38849 (local address 127.0.0.1:50112)
0928 21:09:08.790809 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:08.790823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:08.790906 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:08.791077 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:38849: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 21:09:08.791823 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2763
I20260928 21:09:08.801141 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2393
I20260928 21:09:08.807557 22065 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:09:09.142645 22065 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:09:09.142802 22065 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:09:09.489338 22065 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:09:09.489 UTC [1454] LOG: received smart shutdown request
2026-09-28 21:09:09.494 UTC [1454] LOG: background worker "logical replication launcher" (PID 1463) exited with exit code 1
2026-09-28 21:09:09.494 UTC [1458] LOG: shutting down
2026-09-28 21:09:09.679 UTC [1454] LOG: database system is shut down
2026-09-28T21:09:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50208 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:09:09 dist-test-slave-d5wc krb5kdc[2965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:09:09 dist-test-slave-d5wc krb5kdc[2965](info): set up 2 sockets
Sep 28 21:09:09 dist-test-slave-d5wc krb5kdc[2965](info): commencing operation
krb5kdc: starting...
W20260928 21:09:10.888108 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:09:10 dist-test-slave-d5wc krb5kdc[2965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:09:11.017745 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40575
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40575,127.21.140.125:39679,127.21.140.124:40313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:11.129371 2978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:11.129732 2978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:11.129848 2978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:11.133905 2978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:11.134023 2978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:11.134078 2978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:11.134114 2978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:11.134146 2978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:11.139480 2978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40575,127.21.140.125:39679,127.21.140.124:40313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:11.140599 2978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:11.141710 2978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:11.150504 2983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:11.150520 2978 server_base.cc:1034] running on GCE node
W20260928 21:09:11.150540 2986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:11.150524 2984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:11.151368 2978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:11.152496 2978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:11.153666 2978 hybrid_clock.cc:648] HybridClock initialized: now 1790629751153651 us; error 27 us; skew 500 ppm
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:11.156471 2978 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:11.157413 2978 webserver.cc:460] Webserver started at http://127.21.140.126:46723/ using document root <none> and password file <none>
I20260928 21:09:11.158041 2978 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:11.158134 2978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:11.158373 2978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:11.160398 2978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "7d9ad56ef1164b56884866ec80b9c231"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "249abbc8b2b284d643e0451a7a9b445c"
server_key_iv: "1a273ae30ef292e6e61a7a24b14f1311"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.160975 2978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "7d9ad56ef1164b56884866ec80b9c231"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "249abbc8b2b284d643e0451a7a9b445c"
server_key_iv: "1a273ae30ef292e6e61a7a24b14f1311"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.164793 2978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 21:09:11.167210 2993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:11.168476 2978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:09:11.168643 2978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "7d9ad56ef1164b56884866ec80b9c231"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "249abbc8b2b284d643e0451a7a9b445c"
server_key_iv: "1a273ae30ef292e6e61a7a24b14f1311"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.168784 2978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:11.179440 2978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:11.182229 2978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:11.182436 2978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:11.189203 2978 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40575
I20260928 21:09:11.189518 3046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40575 every 8 connection(s)
I20260928 21:09:11.190193 2978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:11.192858 3047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:11.194515 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 2978
I20260928 21:09:11.194660 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:09:11.194936 22065 external_mini_cluster.cc:1428] Setting key 0eb091e29898aefc69ca6f3050b16e76
I20260928 21:09:11.196702 3047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.140.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.125@KRBTEST.COM" created.
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:11.213011 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.199240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40575 (local address 127.0.0.1:32974)
0928 21:09:11.199799 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.199841 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.200750 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.201447 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.201452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.201881 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.202183 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.202199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.203401 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.203403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.204120 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.204129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.204780 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.204784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.204956 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.205554 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.205570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.207522 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.210018 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.210031 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.210038 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.210261 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.210649 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.210651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.210653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.210792 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.211599 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.211604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.211769 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.211902 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.212147 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":49,"thread_start_us":151,"threads_started":1}
I20260928 21:09:11.213014 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.200269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.21.140.126:40575)
0928 21:09:11.200600 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.200612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.200655 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:09:11.201053 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.201057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.201258 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.201447 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.202347 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.203267 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.204251 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.204615 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.204838 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.207693 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.207700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.207713 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.207764 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.209853 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.210380 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.210388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.210392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.210540 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.210901 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.210906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.210911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.211468 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.211618 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.212044 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.212214 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":4352,"thread_start_us":99,"threads_started":1}
I20260928 21:09:11.215857 3047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:09:11.216626 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.216336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58128)
0928 21:09:11.216494 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.216563 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 21:09:11.216857 2996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
I20260928 21:09:11.217831 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.217554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58140)
0928 21:09:11.217724 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.217777 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 21:09:11.218132 3047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39679: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab.
Entry for principal kudu/127.21.140.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab.
I20260928 21:09:11.222906 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:39679
--webserver_interface=127.21.140.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab
--principal=kudu/127.21.140.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40575,127.21.140.125:39679,127.21.140.124:40313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:11.267417 3047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } attempt: 1
I20260928 21:09:11.268443 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.268045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58154)
0928 21:09:11.268250 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.268328 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 21:09:11.269364 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.269103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58164)
0928 21:09:11.269247 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.269299 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 21:09:11.269498 3047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39679: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
I20260928 21:09:11.337679 3047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } attempt: 2
I20260928 21:09:11.338666 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.338262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58176)
0928 21:09:11.338451 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.338538 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 21:09:11.339464 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.339241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58186)
0928 21:09:11.339354 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.339409 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:09:11.339583 3047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39679: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
W20260928 21:09:11.348992 3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:11.349291 3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:11.349397 3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:11.353508 3054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:11.353605 3054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:11.353642 3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:11.353698 3054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:11.353734 3054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:11.359119 3054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40575,127.21.140.125:39679,127.21.140.124:40313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab
--principal=kudu/127.21.140.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.140.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:11.360179 3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:11.361181 3054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:11.369446 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:11.369441 3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:11.369418 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:11.369705 3054 server_base.cc:1034] running on GCE node
I20260928 21:09:11.370337 3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:11.371450 3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:11.372673 3054 hybrid_clock.cc:648] HybridClock initialized: now 1790629751372638 us; error 46 us; skew 500 ppm
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:11.381371 3054 init.cc:377] Logged in from keytab as kudu/127.21.140.125@KRBTEST.COM (short username kudu)
I20260928 21:09:11.382655 3054 webserver.cc:460] Webserver started at http://127.21.140.125:39313/ using document root <none> and password file <none>
I20260928 21:09:11.383395 3054 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:11.383492 3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:11.383780 3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:11.386675 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data/instance:
uuid: "78b02488eb1d4c42b3081f3f9309511b"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "d154a611a67ed13ffb6edb302addba4a"
server_key_iv: "f85b853d3f679af87d0dedbab1f9a644"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.387204 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal/instance:
uuid: "78b02488eb1d4c42b3081f3f9309511b"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "d154a611a67ed13ffb6edb302addba4a"
server_key_iv: "f85b853d3f679af87d0dedbab1f9a644"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.391098 3054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:09:11.393581 3069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:11.395000 3054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:09:11.395188 3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal
uuid: "78b02488eb1d4c42b3081f3f9309511b"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "d154a611a67ed13ffb6edb302addba4a"
server_key_iv: "f85b853d3f679af87d0dedbab1f9a644"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.395318 3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:11.428252 3054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:11.430725 3047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } attempt: 3
I20260928 21:09:11.431584 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.431283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58194)
0928 21:09:11.431439 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.431507 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 21:09:11.431854 3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:11.432067 3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:11.432309 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.432085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58200)
0928 21:09:11.432200 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.432255 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:09:11.432441 3047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39679: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39679: connect: Connection refused (error 111)
I20260928 21:09:11.439831 3054 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.125:39679
I20260928 21:09:11.440140 3121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.125:39679 every 8 connection(s)
I20260928 21:09:11.440933 3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
I20260928 21:09:11.443688 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:11.447408 3122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:09:11.450011 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3054
I20260928 21:09:11.450156 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-1/wal/instance
I20260928 21:09:11.450448 22065 external_mini_cluster.cc:1428] Setting key fb7e8c3b8c54fb15d144f11a00f79060
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.124@KRBTEST.COM" created.
I20260928 21:09:11.462984 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.449258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.21.140.126:40575)
0928 21:09:11.449418 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.449424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.450522 (+ 1098us) server_negotiation.cc:407] Connection header received
0928 21:09:11.451519 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.451525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.451593 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.451700 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.452897 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.453819 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.455084 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.455538 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.455692 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.458872 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.458875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.458877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.458959 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.460709 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.461230 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.461234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.461236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.461323 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.461767 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.461771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.461772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.462025 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.462148 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.462685 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.462808 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:09:11.463820 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.449755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40575 (local address 127.0.0.1:32986)
0928 21:09:11.450345 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.450399 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.451304 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.451878 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.451885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.452249 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.452724 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.452739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.454035 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.454039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.454955 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.454965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.455745 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.455748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.455964 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.456668 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.456695 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.458653 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.460830 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.460836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.460851 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.461116 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.461456 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.461459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.461461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.461665 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.462139 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.462145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.462425 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.462621 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.462945 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":214,"threads_started":1}
I20260928 21:09:11.465467 3122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:09:11.478508 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.466064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58204)
0928 21:09:11.466244 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.466261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.466340 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.467026 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.467029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.467163 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.467258 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.467263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.469000 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.469003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.469558 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.469567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.470588 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.470591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.470732 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.471270 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.471282 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.474595 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.476747 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.476750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.476752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.476936 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.477243 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.477246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.477247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.477322 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.477866 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.477869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.478052 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.478174 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.478313 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":25}
I20260928 21:09:11.478528 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.466163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.21.140.125:39679)
0928 21:09:11.466523 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.466538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.466565 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:09:11.466628 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.466637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.466884 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.467052 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.467376 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.468857 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.469728 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.470444 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.470582 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.474792 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.474802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.474815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.474835 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.476621 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.477048 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.477059 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.477060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.477144 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.477418 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.477421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.477423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.477751 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.477893 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.478256 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.478379 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":236,"spinlock_wait_cycles":2944,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab.
Entry for principal kudu/127.21.140.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab.
I20260928 21:09:11.480726 3122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:09:11.481544 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.481191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50110)
0928 21:09:11.481344 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.481444 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 21:09:11.481765 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:40313
--webserver_interface=127.21.140.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab
--principal=kudu/127.21.140.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40575,127.21.140.125:39679,127.21.140.124:40313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:11.481792 3073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
I20260928 21:09:11.482857 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.482543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50124)
0928 21:09:11.482698 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.482768 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 21:09:11.483024 3122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:40313: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
I20260928 21:09:11.532333 3122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } attempt: 1
I20260928 21:09:11.533349 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.533033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50126)
0928 21:09:11.533204 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.533269 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 21:09:11.534231 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.533957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50132)
0928 21:09:11.534109 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.534170 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:09:11.534376 3122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:40313: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
I20260928 21:09:11.575618 3047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } attempt: 4
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:09:11.585196 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.576235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58208)
0928 21:09:11.576444 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.576460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.576630 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.576953 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.576957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.577142 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.577248 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.577253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.578074 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.578079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.578624 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.578634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.579593 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.579597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.579709 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.580232 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.580243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.581715 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.583714 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.583717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.583719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.583916 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.584248 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.584250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.584252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.584338 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.584766 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.584768 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.584853 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.584941 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.585050 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 21:09:11.585302 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.576312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.21.140.125:39679)
0928 21:09:11.576466 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.576471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.576558 (+ 87us) server_negotiation.cc:407] Connection header received
0928 21:09:11.576758 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.576762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.576816 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.576910 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.577377 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.577934 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.578751 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.579452 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.579582 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.581873 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.581877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.581880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.581911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.583579 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.584020 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.584024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.584025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.584123 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.584438 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.584442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.584444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.584667 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.584770 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.585017 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.585146 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 21:09:11.586163 3047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:09:11.586884 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.586623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50140)
0928 21:09:11.586751 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.586811 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:09:11.587620 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.587389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50144)
0928 21:09:11.587506 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.587560 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:09:11.587757 3047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:40313: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
W20260928 21:09:11.595525 3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:11.595757 3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:11.595835 3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:11.600050 3129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:11.600129 3129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:11.600167 3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:11.600198 3129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:11.600214 3129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:11.602543 3122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } attempt: 2
I20260928 21:09:11.603299 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.603000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50156)
0928 21:09:11.603170 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.603227 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 21:09:11.604014 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.603804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50160)
0928 21:09:11.603913 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.603966 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:09:11.604154 3122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:40313: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
I20260928 21:09:11.605901 3129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40575,127.21.140.125:39679,127.21.140.124:40313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab
--principal=kudu/127.21.140.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:40313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.140.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:11.606948 3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:11.607985 3129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:09:11.616385 3129 server_base.cc:1034] running on GCE node
W20260928 21:09:11.616374 3134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:11.616417 3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:11.616403 3135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:11.617271 3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:11.618387 3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:11.619606 3129 hybrid_clock.cc:648] HybridClock initialized: now 1790629751619590 us; error 28 us; skew 500 ppm
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:11.622527 3129 init.cc:377] Logged in from keytab as kudu/127.21.140.124@KRBTEST.COM (short username kudu)
I20260928 21:09:11.623459 3129 webserver.cc:460] Webserver started at http://127.21.140.124:39467/ using document root <none> and password file <none>
I20260928 21:09:11.624064 3129 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:11.624168 3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:11.624397 3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:11.626506 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data/instance:
uuid: "b07a75305ddd4965b529d73ef6f05a09"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "ea5749eba2bfa101ff8c94c5dcaf28eb"
server_key_iv: "45d0c6183d3ed69d96bdf2582fb2d912"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.627099 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal/instance:
uuid: "b07a75305ddd4965b529d73ef6f05a09"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "ea5749eba2bfa101ff8c94c5dcaf28eb"
server_key_iv: "45d0c6183d3ed69d96bdf2582fb2d912"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.630926 3129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:09:11.633399 3144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:11.634680 3129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:11.634817 3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal
uuid: "b07a75305ddd4965b529d73ef6f05a09"
format_stamp: "Formatted at 2026-09-28 21:09:11 on dist-test-slave-d5wc"
server_key: "ea5749eba2bfa101ff8c94c5dcaf28eb"
server_key_iv: "45d0c6183d3ed69d96bdf2582fb2d912"
server_key_version: "encryptionkey@0"
I20260928 21:09:11.634929 3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:11.646909 3047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } attempt: 1
I20260928 21:09:11.647709 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.647425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50174)
0928 21:09:11.647572 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.647637 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 21:09:11.648470 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.648252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50188)
0928 21:09:11.648362 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.648414 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 21:09:11.648598 3047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:40313: Network error: Client connection negotiation failed: client connection to 127.21.140.124:40313: connect: Connection refused (error 111)
I20260928 21:09:11.655292 3129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:11.658134 3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:11.658327 3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:11.665606 3129 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.124:40313
I20260928 21:09:11.665927 3196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.124:40313 every 8 connection(s)
I20260928 21:09:11.666612 3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
I20260928 21:09:11.668387 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3129
I20260928 21:09:11.668485 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629583469051-22065-0/minicluster-data/master-2/wal/instance
I20260928 21:09:11.668815 22065 external_mini_cluster.cc:1428] Setting key c07d63c188958b2bd5a6beeff68502c1
I20260928 21:09:11.669586 3197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:11.674283 3197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:11.684012 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.670184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.21.140.124:40313)
0928 21:09:11.670596 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.670614 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.670655 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:09:11.671534 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.671556 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.671958 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.672328 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.673228 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.674719 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.675609 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.676196 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.676375 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.679213 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.679222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.679238 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.679275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.681192 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.681654 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.681660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.681665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.681801 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.682131 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.682135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.682137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.682498 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.682766 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.682940 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.683223 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260928 21:09:11.685361 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.676082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.21.140.126:40575)
0928 21:09:11.676256 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.676260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.676589 (+ 329us) server_negotiation.cc:407] Connection header received
0928 21:09:11.676727 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.676734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.676788 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.676858 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.677769 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.678302 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.679272 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.679475 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.679585 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.682058 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.682061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.682063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.682085 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.683585 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.684071 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.684075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.684077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.684182 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.684474 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.684478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.684480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.684655 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.684777 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.685107 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.685213 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:09:11.685376 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.676041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40575 (local address 127.0.0.1:32994)
0928 21:09:11.676458 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.676506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.676620 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.676947 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.676952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.677426 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.677611 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.677623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.678432 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.678435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.679156 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.679165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.679600 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.679604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.679729 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.680203 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.680215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.681910 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.683702 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.683710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.683716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.683957 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.684301 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.684304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.684305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.684378 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.684776 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.684782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.684890 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.685027 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.685214 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":152,"threads_started":1}
I20260928 21:09:11.686444 3197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:09:11.694849 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.686256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.21.140.126:40575)
0928 21:09:11.686360 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.686364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.686377 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:09:11.686470 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.686474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.686514 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.686595 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.687238 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.687710 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.688463 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.688689 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.688798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.691253 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.691256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.691258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.691299 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.693106 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.693557 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.693562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.693563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.693650 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.693934 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.693938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.693940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.694147 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.694377 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.694548 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.694662 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:09:11.695017 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.687038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.21.140.125:39679)
0928 21:09:11.687177 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.687184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.687212 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:09:11.687253 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.687256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.687302 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.687385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.687761 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.688265 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.688990 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.689354 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.689492 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.691785 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.691789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.691795 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.691828 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.693403 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.693807 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.693812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.693814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.693898 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.694201 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.694205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.694207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.694379 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.694497 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.694758 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.694863 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:09:11.695317 3122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } attempt: 3
I20260928 21:09:11.695640 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.686921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39679 (local address 127.0.0.1:58222)
0928 21:09:11.687048 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.687058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.687143 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.687397 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.687400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.687560 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.687645 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.687651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.688368 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.688374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.688874 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.688881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.689494 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.689498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.689601 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.690077 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.690087 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.691621 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.693508 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.693513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.693515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.693705 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.694015 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.694018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.694020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.694100 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.694498 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.694502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.694593 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.694694 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.694818 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 21:09:11.696866 3197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:09:11.705302 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.695702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50206)
0928 21:09:11.695811 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.695825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.695902 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.696183 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.696185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.696354 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.696469 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.696480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.697639 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.697642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.698527 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.698541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.699169 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.699172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.699309 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.699788 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.699803 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.701237 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.703536 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.703540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.703543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.703753 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.704064 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.704069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.704071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.704181 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.704684 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.704688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.704799 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.704893 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.705103 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":130}
I20260928 21:09:11.705459 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.695749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.21.140.124:40313)
0928 21:09:11.695999 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.696003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.696015 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:09:11.696045 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.696047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.696087 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.696169 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.696591 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.697497 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.698688 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.699036 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.699530 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.701371 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.701635 (+ 264us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.701639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.701679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.703438 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.703862 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.703866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.703867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.703950 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.704313 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.704319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.704324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.704556 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.705089 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.705168 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.705282 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":1280}
I20260928 21:09:11.706606 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.697323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50218)
0928 21:09:11.697471 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.697484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.697554 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.697915 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.697917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.698097 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.698172 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.698178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.699606 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.699610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.700412 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.700425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.701148 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.701152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.701236 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.701671 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.701690 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.703082 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.705007 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.705010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.705012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.705235 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.705541 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.705545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.705549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.705646 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.706060 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.706063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.706166 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.706293 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.706438 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 21:09:11.707340 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.696688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.21.140.125:39679)
0928 21:09:11.696777 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.696780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.696794 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:09:11.696829 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.696832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.696867 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.696948 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.697664 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.698474 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.699342 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.699682 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.699801 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.703335 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.703338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.703341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.703389 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.705209 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.705693 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.705697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.705721 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.705812 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.706146 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.706150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.706151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.706361 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.706592 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.707156 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.707243 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":25}
I20260928 21:09:11.708856 3122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: Bootstrap starting.
I20260928 21:09:11.710330 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.697345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.21.140.124:40313)
0928 21:09:11.697662 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.697670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.697681 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:09:11.697708 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.697710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.697749 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.697828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.698288 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.699497 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.700552 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.701012 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.701138 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.703244 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.703248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.703251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.703310 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.704880 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.705338 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.705343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.705346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.705432 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.705751 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.705757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.705762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.705984 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.706071 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.706356 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.706509 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260928 21:09:11.711382 3122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:11.712309 3122 log.cc:824] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:11.713369 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:09:11.713452 3197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09: Bootstrap starting.
I20260928 21:09:11.714470 3122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: No bootstrap required, opened a new log
I20260928 21:09:11.715442 3197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:11.715783 3047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } attempt: 2
I20260928 21:09:11.716218 3197 log.cc:824] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:11.718727 3122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.718957 3122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:11.719012 3122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b02488eb1d4c42b3081f3f9309511b, State: Initialized, Role: FOLLOWER
I20260928 21:09:11.719504 3122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.720312 3197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09: No bootstrap required, opened a new log
I20260928 21:09:11.721185 3122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:11.724619 3197 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.724858 3197 raft_consensus.cc:374] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:11.724918 3197 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07a75305ddd4965b529d73ef6f05a09, State: Initialized, Role: FOLLOWER
Sep 28 21:09:11 dist-test-slave-d5wc krb5kdc[2965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629751, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:09:11.726785 3212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.726922 3212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:09:11.726874 3197 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.728236 3197 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:11.728716 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.716668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.21.140.125:39679)
0928 21:09:11.716776 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.716779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.719207 (+ 2428us) server_negotiation.cc:407] Connection header received
0928 21:09:11.719247 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.719250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.719291 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.719358 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.720272 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.720985 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.722113 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.722425 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.722537 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.724281 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.724288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.724292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.724320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.726237 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.726713 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.726718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.726720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.726810 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.727529 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.727534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.727537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.727775 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.727910 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.728429 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.728539 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:09:11.730062 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.716154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:40313 (local address 127.0.0.1:50240)
0928 21:09:11.716335 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:11.716350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:11.716431 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:11.716928 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:11.716931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:11.717144 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:11.717247 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.717254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.718283 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.718286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.719085 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:11.719095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.719817 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.719820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:11.719904 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.721210 (+ 1306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:11.721226 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:11.726157 (+ 4931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:11.728333 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.728338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.728340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.728593 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.728929 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:11.728932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:11.728934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:11.729030 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:11.729509 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:11.729512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:11.729619 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:09:11.729737 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 21:09:11.729880 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 21:09:11.730062 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.716614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.21.140.124:40313)
0928 21:09:11.716704 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.716709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.716723 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:09:11.716758 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.716761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.716805 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.716907 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.717379 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.718172 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.719194 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.719496 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.719598 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.726314 (+ 6716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.726318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.726320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.726344 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.728212 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.728722 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.728728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.728733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.728827 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.729141 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.729146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.729148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.729371 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.729463 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.729789 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.729895 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:09:11.725000 3054 master_runner.cc:186] Error getting master registration for 127.21.140.126:40575: OK, instance_id { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" instance_seqno: 1790629751187351 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:09:11.732817 3054 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:09:11.733187 3047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: Bootstrap starting.
I20260928 21:09:11.736150 3047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:11.737170 3047 log.cc:824] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:11.738175 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.717440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.21.140.126:40575)
0928 21:09:11.717571 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.717575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.720434 (+ 2859us) server_negotiation.cc:407] Connection header received
0928 21:09:11.720611 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.720616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.720668 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.720733 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.723287 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.724035 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.725703 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.726058 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.726141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.734051 (+ 7910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.734055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.734058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.734088 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.735893 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.736487 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.736491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.736492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.736587 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.736884 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.736887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.736889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.737091 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.737226 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.737859 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.738007 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 21:09:11.739146 3223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:09:11.739238 3223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:09:11.735313 3129 master_runner.cc:186] Error getting master registration for 127.21.140.126:40575: OK, instance_id { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" instance_seqno: 1790629751187351 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:09:11.739547 3129 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:09:11.739711 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:11.715858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.21.140.124:40313)
0928 21:09:11.716381 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:11.716386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:11.718457 (+ 2071us) server_negotiation.cc:407] Connection header received
0928 21:09:11.718619 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:11.718622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:11.718664 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:11.718710 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:11.731075 (+ 12365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.731837 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.733026 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:11.733376 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:11.733452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:11.735734 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:11.735738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:11.735741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:11.735766 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:11.737570 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.738098 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.738101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.738103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.738216 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:11.738567 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:11.738571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:11.738573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:11.738804 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:11.738943 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:11.739352 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 21:09:11.739502 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260928 21:09:11.741156 3047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: No bootstrap required, opened a new log
I20260928 21:09:11.742278 3224 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.742413 3224 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:09:11.745548 3047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.745791 3047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:11.745853 3047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9ad56ef1164b56884866ec80b9c231, State: Initialized, Role: FOLLOWER
I20260928 21:09:11.746376 3047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.748309 3047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:09:11.750291 3236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:09:11.750373 3236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:09:11.750988 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.751119 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:09:11.753557 3250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:09:11.753646 3250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:09:11.757014 2978 master_runner.cc:263] No leader master found from master 7d9ad56ef1164b56884866ec80b9c231
I20260928 21:09:11.757102 2978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:09:11.794039 3224 raft_consensus.cc:482] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:11.794238 3224 raft_consensus.cc:504] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.795250 3224 leader_election.cc:288] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d9ad56ef1164b56884866ec80b9c231 (127.21.140.126:40575), 78b02488eb1d4c42b3081f3f9309511b (127.21.140.125:39679)
I20260928 21:09:11.795871 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b07a75305ddd4965b529d73ef6f05a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b02488eb1d4c42b3081f3f9309511b" is_pre_election: true
I20260928 21:09:11.796190 3096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07a75305ddd4965b529d73ef6f05a09 in term 0.
I20260928 21:09:11.796622 3147 leader_election.cc:302] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b02488eb1d4c42b3081f3f9309511b, b07a75305ddd4965b529d73ef6f05a09; no voters:
I20260928 21:09:11.797353 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b07a75305ddd4965b529d73ef6f05a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9ad56ef1164b56884866ec80b9c231" is_pre_election: true
I20260928 21:09:11.797662 3021 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07a75305ddd4965b529d73ef6f05a09 in term 0.
I20260928 21:09:11.798231 3224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:09:11.798326 3224 raft_consensus.cc:482] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:09:11.798386 3224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:11.799748 3224 raft_consensus.cc:504] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.800079 3224 leader_election.cc:288] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [CANDIDATE]: Term 1 election: Requested vote from peers 7d9ad56ef1164b56884866ec80b9c231 (127.21.140.126:40575), 78b02488eb1d4c42b3081f3f9309511b (127.21.140.125:39679)
I20260928 21:09:11.800504 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b07a75305ddd4965b529d73ef6f05a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b02488eb1d4c42b3081f3f9309511b"
I20260928 21:09:11.800684 3096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:11.802269 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b07a75305ddd4965b529d73ef6f05a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9ad56ef1164b56884866ec80b9c231"
I20260928 21:09:11.802394 3021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:11.803618 3021 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b07a75305ddd4965b529d73ef6f05a09 in term 1.
I20260928 21:09:11.803942 3146 leader_election.cc:302] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9ad56ef1164b56884866ec80b9c231, b07a75305ddd4965b529d73ef6f05a09; no voters:
I20260928 21:09:11.801959 3096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b07a75305ddd4965b529d73ef6f05a09 in term 1.
I20260928 21:09:11.807492 3224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:11.807739 3224 raft_consensus.cc:686] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [term 1 LEADER]: Becoming Leader. State: Replica: b07a75305ddd4965b529d73ef6f05a09, State: Running, Role: LEADER
I20260928 21:09:11.808089 3224 consensus_queue.cc:237] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } }
I20260928 21:09:11.809795 3224 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07a75305ddd4965b529d73ef6f05a09. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.809922 3224 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:11.810338 3254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:11.811255 3254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:11.819949 3096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [term 1 FOLLOWER]: Refusing update from remote peer b07a75305ddd4965b529d73ef6f05a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:11.821015 3021 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [term 1 FOLLOWER]: Refusing update from remote peer b07a75305ddd4965b529d73ef6f05a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:11.823385 3224 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:11.823840 3224 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:11.827767 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07a75305ddd4965b529d73ef6f05a09. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.827905 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:09:11.831291 3212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07a75305ddd4965b529d73ef6f05a09. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.831415 3212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:09:11.839550 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.839691 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:11.840794 3212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.840912 3212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:09:11.841436 3258 mvcc.cc:205] Tried to move back new op lower bound from 7334419463444078592 to 7334419463409029120. Current Snapshot: MvccSnapshot[applied={T|T < 7334419463444078592}]
I20260928 21:09:11.843225 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.843359 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:09:11.842126 3224 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b07a75305ddd4965b529d73ef6f05a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9ad56ef1164b56884866ec80b9c231" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40575 } } peers { permanent_uuid: "78b02488eb1d4c42b3081f3f9309511b" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39679 } } peers { permanent_uuid: "b07a75305ddd4965b529d73ef6f05a09" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 40313 } } }
I20260928 21:09:11.843822 3224 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:11.845186 3256 mvcc.cc:205] Tried to move back new op lower bound from 7334419463444078592 to 7334419463409029120. Current Snapshot: MvccSnapshot[applied={T|T < 7334419463444078592}]
I20260928 21:09:11.847489 3254 catalog_manager.cc:1347] Generated new cluster ID: 1f8a455be1194fd9b40c025a60fa3277
I20260928 21:09:11.847558 3254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:11.870510 3254 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:09:11.871712 3254 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:11.894508 3254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b07a75305ddd4965b529d73ef6f05a09: Generated new TSK 0
I20260928 21:09:11.895376 3254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:09:12.741182 3223 catalog_manager.cc:1259] Loaded cluster ID: 1f8a455be1194fd9b40c025a60fa3277
I20260928 21:09:12.741318 3223 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: loading cluster ID for follower catalog manager: success
I20260928 21:09:12.743708 3223 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: acquiring CA information for follower catalog manager: success
I20260928 21:09:12.745672 3223 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78b02488eb1d4c42b3081f3f9309511b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:09:12.755594 3250 catalog_manager.cc:1259] Loaded cluster ID: 1f8a455be1194fd9b40c025a60fa3277
I20260928 21:09:12.755721 3250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: loading cluster ID for follower catalog manager: success
I20260928 21:09:12.758150 3250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: acquiring CA information for follower catalog manager: success
I20260928 21:09:12.759950 3250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d9ad56ef1164b56884866ec80b9c231: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:09:12.971731 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 2978
I20260928 21:09:12.979985 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3054
I20260928 21:09:12.987855 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3129
2026-09-28T21:09:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3186 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:09:13 dist-test-slave-d5wc krb5kdc[3273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:09:13 dist-test-slave-d5wc krb5kdc[3273](info): set up 2 sockets
Sep 28 21:09:13 dist-test-slave-d5wc krb5kdc[3273](info): commencing operation
krb5kdc: starting...
W20260928 21:09:14.059823 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:09:14 dist-test-slave-d5wc krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:09:14.185312 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33453
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:14.296559 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:14.296823 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:14.296865 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:14.301402 3286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:14.301467 3286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:14.301487 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:14.301537 3286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:14.301576 3286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:14.306792 3286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:33453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:14.307821 3286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:14.308848 3286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:14.317332 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:14.317365 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:14.317334 3286 server_base.cc:1034] running on GCE node
W20260928 21:09:14.317369 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:14.318136 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:14.319226 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:14.320423 3286 hybrid_clock.cc:648] HybridClock initialized: now 1790629754320408 us; error 33 us; skew 500 ppm
Sep 28 21:09:14 dist-test-slave-d5wc krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629754, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:14.323227 3286 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:14.324067 3286 webserver.cc:460] Webserver started at http://127.21.140.126:44547/ using document root <none> and password file <none>
I20260928 21:09:14.324613 3286 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:14.324672 3286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:14.324885 3286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:14.327008 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e"
format_stamp: "Formatted at 2026-09-28 21:09:14 on dist-test-slave-d5wc"
server_key: "8e04d31daccf39f4c471f08ccb546fab"
server_key_iv: "f2be5b654039ec539030ec033fcd6bad"
server_key_version: "encryptionkey@0"
I20260928 21:09:14.327569 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e"
format_stamp: "Formatted at 2026-09-28 21:09:14 on dist-test-slave-d5wc"
server_key: "8e04d31daccf39f4c471f08ccb546fab"
server_key_iv: "f2be5b654039ec539030ec033fcd6bad"
server_key_version: "encryptionkey@0"
I20260928 21:09:14.331367 3286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:09:14.333794 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:14.335039 3286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:14.335178 3286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e"
format_stamp: "Formatted at 2026-09-28 21:09:14 on dist-test-slave-d5wc"
server_key: "8e04d31daccf39f4c471f08ccb546fab"
server_key_iv: "f2be5b654039ec539030ec033fcd6bad"
server_key_version: "encryptionkey@0"
I20260928 21:09:14.335292 3286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:14.350466 3286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:14.353578 3286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:14.353788 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:14.360797 3286 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:33453
I20260928 21:09:14.360813 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:33453 every 8 connection(s)
I20260928 21:09:14.361785 3286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:14.364643 3354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:14.370345 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e: Bootstrap starting.
I20260928 21:09:14.371557 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3286
I20260928 21:09:14.371697 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:09:14.371959 22065 external_mini_cluster.cc:1428] Setting key a42ef93786e513deee5bdaa6e17e4581
I20260928 21:09:14.372906 3354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:14.373728 3354 log.cc:824] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:14.375591 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e: No bootstrap required, opened a new log
I20260928 21:09:14.379086 3354 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33453 } }
I20260928 21:09:14.379343 3354 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:14.379429 3354 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec81f8915c4f4c98b0f28fc6e1ca8d8e, State: Initialized, Role: FOLLOWER
I20260928 21:09:14.379882 3354 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33453 } }
I20260928 21:09:14.380028 3354 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:14.380107 3354 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:14.380213 3354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:09:14 dist-test-slave-d5wc krb5kdc[3273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:14.381181 3354 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33453 } }
I20260928 21:09:14.381548 3354 leader_election.cc:302] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec81f8915c4f4c98b0f28fc6e1ca8d8e; no voters:
I20260928 21:09:14.381911 3354 leader_election.cc:288] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:09:14.382004 3359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:14.382503 3359 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [term 1 LEADER]: Becoming Leader. State: Replica: ec81f8915c4f4c98b0f28fc6e1ca8d8e, State: Running, Role: LEADER
I20260928 21:09:14.382812 3359 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33453 } }
I20260928 21:09:14.383246 3354 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:14.383975 3359 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec81f8915c4f4c98b0f28fc6e1ca8d8e. Latest consensus state: current_term: 1 leader_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33453 } } }
I20260928 21:09:14.384158 3359 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:14.384817 3360 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81f8915c4f4c98b0f28fc6e1ca8d8e" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33453 } } }
I20260928 21:09:14.385003 3360 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:14.387331 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:14.373010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.21.140.126:33453)
0928 21:09:14.373451 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:14.373467 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:14.373502 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:09:14.374244 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:14.374263 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:14.374572 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:14.374913 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:14.375799 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:14.376833 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:14.377724 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:14.378274 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:14.378459 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:14.381148 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:14.381154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:14.381167 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:14.381194 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:14.383141 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:14.383629 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:14.383635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:14.383640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:14.383785 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:14.384130 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:14.384236 (+ 106us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:14.384239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:14.384758 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:14.384945 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:14.386105 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0928 21:09:14.386400 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
W20260928 21:09:14.388552 3375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:09:14.388681 3375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:09:14.388798 3368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:14.389704 3368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:14.392457 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:09:14.395685 3368 catalog_manager.cc:1347] Generated new cluster ID: d2295bc40438430494b9d3bf6e02fc30
I20260928 21:09:14.395776 3368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:14.414176 3368 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:09:14.415216 3368 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:14.426302 3368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec81f8915c4f4c98b0f28fc6e1ca8d8e: Generated new TSK 0
I20260928 21:09:14.427049 3368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:09:14.448405 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3286
2026-09-28T21:09:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1456 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:09:14 dist-test-slave-d5wc krb5kdc[3382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:09:14 dist-test-slave-d5wc krb5kdc[3382](info): set up 2 sockets
Sep 28 21:09:14 dist-test-slave-d5wc krb5kdc[3382](info): commencing operation
krb5kdc: starting...
W20260928 21:09:15.487684 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:09:15 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:09:15.608721 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44961
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:15.718014 3395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:15.718317 3395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:15.718394 3395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:15.722401 3395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:15.722491 3395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:15.722528 3395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:15.722566 3395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:15.722604 3395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:15.727804 3395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:15.728852 3395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:15.729846 3395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:15.738399 3403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:15.738430 3400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:15.738637 3395 server_base.cc:1034] running on GCE node
W20260928 21:09:15.738435 3401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:15.739308 3395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:15.740438 3395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:15.741600 3395 hybrid_clock.cc:648] HybridClock initialized: now 1790629755741570 us; error 49 us; skew 500 ppm
Sep 28 21:09:15 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:15.744441 3395 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:15.745347 3395 webserver.cc:460] Webserver started at http://127.21.140.126:43977/ using document root <none> and password file <none>
I20260928 21:09:15.745910 3395 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:15.746021 3395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:15.746253 3395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:15.748332 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "8ff579ba05c445f8a2552f525dfa6467"
format_stamp: "Formatted at 2026-09-28 21:09:15 on dist-test-slave-d5wc"
server_key: "a6f4b052bf43784c1580cedaa3a99e1c"
server_key_iv: "e5a582855739658c4dfdfd22dd461503"
server_key_version: "encryptionkey@0"
I20260928 21:09:15.748929 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "8ff579ba05c445f8a2552f525dfa6467"
format_stamp: "Formatted at 2026-09-28 21:09:15 on dist-test-slave-d5wc"
server_key: "a6f4b052bf43784c1580cedaa3a99e1c"
server_key_iv: "e5a582855739658c4dfdfd22dd461503"
server_key_version: "encryptionkey@0"
I20260928 21:09:15.752910 3395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 21:09:15.755472 3410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:15.756867 3395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:09:15.757030 3395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "8ff579ba05c445f8a2552f525dfa6467"
format_stamp: "Formatted at 2026-09-28 21:09:15 on dist-test-slave-d5wc"
server_key: "a6f4b052bf43784c1580cedaa3a99e1c"
server_key_iv: "e5a582855739658c4dfdfd22dd461503"
server_key_version: "encryptionkey@0"
I20260928 21:09:15.757175 3395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:15.773249 3395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:15.776510 3395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:15.776749 3395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:15.783653 3395 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44961
I20260928 21:09:15.783924 3462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44961 every 8 connection(s)
I20260928 21:09:15.784809 3395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:15.787801 3463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:15.793208 3463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap starting.
I20260928 21:09:15.794683 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3395
I20260928 21:09:15.794842 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:09:15.795131 22065 external_mini_cluster.cc:1428] Setting key 8cde9a78956952663faae4f08983b436
I20260928 21:09:15.795758 3463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:15.796648 3463 log.cc:824] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:15.798700 3463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: No bootstrap required, opened a new log
I20260928 21:09:15.802208 3463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:15.802491 3463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:15.802578 3463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff579ba05c445f8a2552f525dfa6467, State: Initialized, Role: FOLLOWER
I20260928 21:09:15.803016 3463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:15.803164 3463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:15.803244 3463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:15.803345 3463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:09:15 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:15.804563 3463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:15.804939 3463 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff579ba05c445f8a2552f525dfa6467; no voters:
I20260928 21:09:15.805311 3463 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:09:15.805446 3469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:15.805683 3469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff579ba05c445f8a2552f525dfa6467, State: Running, Role: LEADER
I20260928 21:09:15.806028 3469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:15.806622 3463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:15.807313 3470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff579ba05c445f8a2552f525dfa6467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } } }
I20260928 21:09:15.807519 3470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:15.807708 3471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff579ba05c445f8a2552f525dfa6467. Latest consensus state: current_term: 1 leader_uuid: "8ff579ba05c445f8a2552f525dfa6467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } } }
I20260928 21:09:15.807865 3471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:15.809633 3478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:15.810776 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:15.796228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.21.140.126:44961)
0928 21:09:15.796674 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:15.796688 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:15.796723 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:09:15.797577 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:15.797597 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:15.797921 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:15.798307 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:15.799134 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:15.800096 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:15.800986 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:15.801477 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:15.801659 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:15.804630 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:15.804636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:15.804649 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:15.804677 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:15.806788 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:15.807352 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:15.807360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:15.807368 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:15.807525 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:15.807908 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:15.807912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:15.807915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:15.808487 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:15.808694 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:15.809528 (+ 834us) server_negotiation.cc:299] Negotiation successful
0928 21:09:15.809838 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260928 21:09:15.812615 3478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:15.818367 3478 catalog_manager.cc:1347] Generated new cluster ID: 3170d96bd561400bb3165092940e18f7
I20260928 21:09:15.818501 3478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:15.837543 3478 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:09:15.838639 3478 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:15.844132 3478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Generated new TSK 0
I20260928 21:09:15.844903 3478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:09:15.872381 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44961
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:15.983286 3490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:15.983539 3490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:15.983611 3490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:15.987749 3490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:15.987810 3490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:15.987898 3490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:15.993161 3490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:15.994216 3490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:15.995260 3490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:16.005008 3496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:16.005048 3495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:16.005048 3498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:16.005442 3490 server_base.cc:1034] running on GCE node
I20260928 21:09:16.005859 3490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:16.006636 3490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:16.007830 3490 hybrid_clock.cc:648] HybridClock initialized: now 1790629756007803 us; error 30 us; skew 500 ppm
Sep 28 21:09:16 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:16.010883 3490 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:16.011811 3490 webserver.cc:460] Webserver started at http://127.21.140.65:46599/ using document root <none> and password file <none>
I20260928 21:09:16.012413 3490 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:16.012480 3490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:16.012701 3490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:16.014827 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "5c6a11c5b2114197a6ad5d432b7be92b"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "b22d00604bfdd86f4eb51c9f7a9e4367"
server_key_iv: "ac8046855ab00f6195d0e57c714d8cfa"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.015398 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "5c6a11c5b2114197a6ad5d432b7be92b"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "b22d00604bfdd86f4eb51c9f7a9e4367"
server_key_iv: "ac8046855ab00f6195d0e57c714d8cfa"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.019166 3490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 21:09:16.021589 3505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.022891 3490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:16.023051 3490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "5c6a11c5b2114197a6ad5d432b7be92b"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "b22d00604bfdd86f4eb51c9f7a9e4367"
server_key_iv: "ac8046855ab00f6195d0e57c714d8cfa"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.023166 3490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:16.054697 3490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:16.057869 3490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:16.058110 3490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:16.058741 3490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:16.059580 3490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:16.059649 3490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.059719 3490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:16.059768 3490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.068773 3490 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:32919
I20260928 21:09:16.068840 3619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:32919 every 8 connection(s)
I20260928 21:09:16.069703 3490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:16.078997 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3490
I20260928 21:09:16.079138 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
Sep 28 21:09:16 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:16.079478 22065 external_mini_cluster.cc:1428] Setting key 98072a4a61d7f245649f36b550b4694d
I20260928 21:09:16.084163 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.071564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:41551 (local address 127.21.140.126:44961)
0928 21:09:16.071711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.071714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.072473 (+ 759us) server_negotiation.cc:407] Connection header received
0928 21:09:16.073913 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.073917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.074000 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.074100 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:16.075282 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.075836 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.076977 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.077218 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.077326 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.080118 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:16.080121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:16.080123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:16.080149 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:16.081732 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.082422 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.082425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.082426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.082502 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.082915 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.082920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.082923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.083174 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:16.083281 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.083768 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.083921 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:09:16.084822 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.071851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:41551)
0928 21:09:16.072321 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.072365 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.073700 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.074283 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.074290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.074657 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:16.075134 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.075148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.075963 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.075972 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.076850 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.076859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.077377 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.077382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.077543 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.078179 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:16.078195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:16.079932 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:16.081855 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:16.081865 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:16.081884 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:16.082325 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:16.082593 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:16.082599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:16.082604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:16.082801 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:16.083280 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:16.083284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:16.083519 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 21:09:16.083681 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 21:09:16.083945 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260928 21:09:16.086122 3620 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44961
I20260928 21:09:16.086369 3620 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:16.086848 3620 heartbeater.cc:511] Master 127.21.140.126:44961 requested a full tablet report, sending...
I20260928 21:09:16.088308 3427 ts_manager.cc:194] Registered new tserver with Master: 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919)
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:09:16.089468 3427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:41551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:09:16.105587 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44961
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:16.216169 3625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:16.216476 3625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:16.216548 3625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:16.220690 3625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:16.220781 3625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:16.220871 3625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:16.226292 3625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:16.227393 3625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:16.228403 3625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:09:16.238376 3625 server_base.cc:1034] running on GCE node
W20260928 21:09:16.238317 3631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:16.238394 3633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:16.238317 3630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:16.239077 3625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:16.239693 3625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:16.240883 3625 hybrid_clock.cc:648] HybridClock initialized: now 1790629756240872 us; error 31 us; skew 500 ppm
Sep 28 21:09:16 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:16.243619 3625 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:16.244405 3625 webserver.cc:460] Webserver started at http://127.21.140.66:40721/ using document root <none> and password file <none>
I20260928 21:09:16.244910 3625 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:16.244966 3625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:16.245121 3625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:16.247093 3625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "507e5ac2fbd14615aea7686c67ba5c16"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "cc2071e5168687cf1f5ae21865e9205f"
server_key_iv: "c265f6722e11537526a82ae8e82d114a"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.247597 3625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "507e5ac2fbd14615aea7686c67ba5c16"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "cc2071e5168687cf1f5ae21865e9205f"
server_key_iv: "c265f6722e11537526a82ae8e82d114a"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.251214 3625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:09:16.253604 3640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.254806 3625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:16.254900 3625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "507e5ac2fbd14615aea7686c67ba5c16"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "cc2071e5168687cf1f5ae21865e9205f"
server_key_iv: "c265f6722e11537526a82ae8e82d114a"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.255009 3625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:16.292534 3625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:16.295838 3625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:16.296039 3625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:16.296622 3625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:16.297464 3625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:16.297544 3625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.297609 3625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:16.297650 3625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.306205 3625 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:35281
I20260928 21:09:16.306222 3754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:35281 every 8 connection(s)
I20260928 21:09:16.307179 3625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:09:16.312371 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3625
I20260928 21:09:16.312467 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:09:16.312749 22065 external_mini_cluster.cc:1428] Setting key e60a5bcf3cacade53570c8324fc30a75
Sep 28 21:09:16 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:16.321626 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.309075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:56405 (local address 127.21.140.126:44961)
0928 21:09:16.309236 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.309240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.310080 (+ 840us) server_negotiation.cc:407] Connection header received
0928 21:09:16.311440 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.311444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.311500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.311604 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:16.312824 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.313463 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.314700 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.314919 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.315026 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.317678 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:16.317685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:16.317692 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:16.317734 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:16.319481 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.319982 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.319985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.319987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.320083 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.320421 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.320424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.320426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.320681 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:16.320803 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.321297 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.321423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:09:16.322438 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.309446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:56405)
0928 21:09:16.309906 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.309947 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.311236 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.311778 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.311784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.312135 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:16.312680 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.312696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.313592 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.313596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.314572 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.314582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.315063 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.315067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.315232 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.315861 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:16.315886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:16.317511 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:16.319605 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:16.319610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:16.319621 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:16.319880 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:16.320180 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:16.320182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:16.320184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:16.320326 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:16.320806 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:16.320813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:16.321056 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 21:09:16.321211 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 21:09:16.321475 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260928 21:09:16.323619 3755 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44961
I20260928 21:09:16.324041 3755 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:16.324648 3755 heartbeater.cc:511] Master 127.21.140.126:44961 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:09:16.325845 3427 ts_manager.cc:194] Registered new tserver with Master: 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281)
I20260928 21:09:16.326783 3427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:56405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:09:16.340988 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44961
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:16.450344 3760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:16.450613 3760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:16.450656 3760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:16.454849 3760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:16.454906 3760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:16.454993 3760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:16.460382 3760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:16.461380 3760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:16.462440 3760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:16.472045 3766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:16.472014 3768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:16.472085 3765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:16.472034 3760 server_base.cc:1034] running on GCE node
I20260928 21:09:16.472697 3760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:16.473361 3760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:16.474535 3760 hybrid_clock.cc:648] HybridClock initialized: now 1790629756474507 us; error 50 us; skew 500 ppm
Sep 28 21:09:16 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:16.484282 3760 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:16.485249 3760 webserver.cc:460] Webserver started at http://127.21.140.67:33837/ using document root <none> and password file <none>
I20260928 21:09:16.485821 3760 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:16.485927 3760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:16.486186 3760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:16.488300 3760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "5d36e1916f704be691697c9e2851f5f7"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "fc39af8b94609231057e55c162f4c17b"
server_key_iv: "ebd8483ed72eb63c7bfa747a11f984d5"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.488862 3760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "5d36e1916f704be691697c9e2851f5f7"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "fc39af8b94609231057e55c162f4c17b"
server_key_iv: "ebd8483ed72eb63c7bfa747a11f984d5"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.492555 3760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 21:09:16.494897 3775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.496013 3760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:09:16.496141 3760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "5d36e1916f704be691697c9e2851f5f7"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "fc39af8b94609231057e55c162f4c17b"
server_key_iv: "ebd8483ed72eb63c7bfa747a11f984d5"
server_key_version: "encryptionkey@0"
I20260928 21:09:16.496261 3760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:16.516631 3760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:16.519618 3760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:16.519821 3760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:16.520397 3760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:16.521231 3760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:16.521301 3760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.521373 3760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:16.521414 3760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:16.530555 3760 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35783
I20260928 21:09:16.530577 3888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35783 every 8 connection(s)
I20260928 21:09:16.531404 3760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:09:16.537310 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3760
I20260928 21:09:16.537411 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:09:16.537694 22065 external_mini_cluster.cc:1428] Setting key d61385a1be4ab81b2f547feb48deeb51
Sep 28 21:09:16 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:16.545058 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.533099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:36457 (local address 127.21.140.126:44961)
0928 21:09:16.533236 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.533239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.534002 (+ 763us) server_negotiation.cc:407] Connection header received
0928 21:09:16.535436 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.535439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.535524 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.535627 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:16.536762 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.537320 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.538751 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.539050 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.539162 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.541443 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:16.541447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:16.541449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:16.541475 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:16.542825 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.543495 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.543499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.543501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.543592 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.543942 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.543947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.543949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.544146 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:16.544253 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.544773 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.544897 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:09:16.545848 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.533421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:36457)
0928 21:09:16.533842 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.533883 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.535222 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.535793 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.535800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.536149 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:16.536614 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.536626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.537502 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.537508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.538624 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.538635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.539195 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.539199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.539374 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.539892 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:16.539907 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:16.541285 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:16.542970 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:16.542976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:16.542990 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:16.543344 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:16.543696 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:16.543699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:16.543701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:16.543844 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:16.544250 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:16.544257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:16.544490 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 21:09:16.544726 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 21:09:16.545077 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260928 21:09:16.547000 3889 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44961
I20260928 21:09:16.547233 3889 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:16.547698 3889 heartbeater.cc:511] Master 127.21.140.126:44961 requested a full tablet report, sending...
I20260928 21:09:16.548655 3427 ts_manager.cc:194] Registered new tserver with Master: 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
I20260928 21:09:16.549223 3427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:36457
I20260928 21:09:16.551190 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:09:16.560055 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.553013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.21.140.126:44961)
0928 21:09:16.553143 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.553149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.553330 (+ 181us) server_negotiation.cc:407] Connection header received
0928 21:09:16.553466 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.553469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.553512 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.553588 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:16.554396 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.554891 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.555699 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.555922 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.556028 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.557289 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:16.557293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:16.557296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:16.557321 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:16.558567 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.558988 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.558992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.558994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.559073 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:16.559369 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:16.559371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:16.559372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:16.559566 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:16.559670 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.559857 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.559948 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:09:16.562599 3427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:09:16.566015 3427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:09:16.579859 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.574249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:32919 (local address 127.0.0.1:33462)
0928 21:09:16.575081 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.575093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.575182 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.576039 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.576045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.576067 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:16.576175 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.576181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.577499 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.577505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.578503 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.578512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.579321 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.579325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.579429 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.579464 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:09:16.579569 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:09:16.579694 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":108,"threads_started":1}
I20260928 21:09:16.580054 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.574057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:35281 (local address 127.0.0.1:38866)
0928 21:09:16.574697 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.574748 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.574867 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.575622 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.575627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.575649 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:16.575783 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.575789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.577229 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.577235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.578491 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.578500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.579566 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.579569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.579694 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.579714 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:09:16.579784 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:09:16.579839 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260928 21:09:16.580235 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.574231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.21.140.66:35281)
0928 21:09:16.575086 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.575098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.575121 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:09:16.575199 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.575206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.575418 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.575618 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:16.575898 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.577088 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.578608 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.579420 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.579558 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.579685 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.579936 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.580052 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":129,"threads_started":1}
I20260928 21:09:16.580775 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.574720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.21.140.65:32919)
0928 21:09:16.575603 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.575619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.575639 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:09:16.575713 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.575718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.575898 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.576049 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:16.576294 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.577368 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.578642 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.579195 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.579317 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.579461 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.580526 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.580625 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":130,"threads_started":1}
I20260928 21:09:16.581401 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.574816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35783 (local address 127.0.0.1:38564)
0928 21:09:16.575365 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.575380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.575458 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.576135 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.576138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.576154 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:16.576228 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.576236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.577314 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.577318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.579675 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.579685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.581004 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.581008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:16.581116 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.581141 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:09:16.581221 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:09:16.581285 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":1201,"thread_start_us":101,"threads_started":1}
I20260928 21:09:16.581499 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.575244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.21.140.67:35783)
0928 21:09:16.575673 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.575688 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.575710 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:09:16.575779 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.575784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.575967 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.576364 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:16.576474 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.577178 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.579786 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.580658 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.580763 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:16.580884 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:16.581276 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 21:09:16.581370 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":143,"threads_started":1}
I20260928 21:09:16.583271 3688 tablet_service.cc:1467] Processing CreateTablet for tablet 58db553e7fe94fa5877fe4a16658549c (DEFAULT_TABLE table=test-table [id=ac352edcdbf2408c903c30dfc0c84d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:16.583503 3553 tablet_service.cc:1467] Processing CreateTablet for tablet 58db553e7fe94fa5877fe4a16658549c (DEFAULT_TABLE table=test-table [id=ac352edcdbf2408c903c30dfc0c84d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:16.584347 3553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58db553e7fe94fa5877fe4a16658549c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:16.584374 3688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58db553e7fe94fa5877fe4a16658549c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:16.584389 3823 tablet_service.cc:1467] Processing CreateTablet for tablet 58db553e7fe94fa5877fe4a16658549c (DEFAULT_TABLE table=test-table [id=ac352edcdbf2408c903c30dfc0c84d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:16.585265 3823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58db553e7fe94fa5877fe4a16658549c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:16.589715 3906 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap starting.
I20260928 21:09:16.591200 3907 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Bootstrap starting.
I20260928 21:09:16.592051 3908 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Bootstrap starting.
I20260928 21:09:16.593215 3906 tablet_bootstrap.cc:654] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:16.593652 3907 tablet_bootstrap.cc:654] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:16.593936 3908 tablet_bootstrap.cc:654] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:16.594163 3906 log.cc:824] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:16.594470 3907 log.cc:824] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:16.595151 3908 log.cc:824] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:16.595950 3906 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: No bootstrap required, opened a new log
I20260928 21:09:16.596144 3906 ts_tablet_manager.cc:1397] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:09:16.596772 3908 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: No bootstrap required, opened a new log
I20260928 21:09:16.596953 3908 ts_tablet_manager.cc:1397] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:09:16.599794 3907 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: No bootstrap required, opened a new log
I20260928 21:09:16.599982 3907 ts_tablet_manager.cc:1397] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 21:09:16.599936 3906 raft_consensus.cc:348] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.600216 3906 raft_consensus.cc:374] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:16.600288 3906 raft_consensus.cc:729] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6a11c5b2114197a6ad5d432b7be92b, State: Initialized, Role: FOLLOWER
I20260928 21:09:16.600427 3908 raft_consensus.cc:348] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.600690 3908 raft_consensus.cc:374] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:16.600740 3908 raft_consensus.cc:729] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d36e1916f704be691697c9e2851f5f7, State: Initialized, Role: FOLLOWER
I20260928 21:09:16.600765 3906 consensus_queue.cc:260] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.601181 3908 consensus_queue.cc:260] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.601394 3620 heartbeater.cc:503] Master 127.21.140.126:44961 was elected leader, sending a full tablet report...
I20260928 21:09:16.601994 3906 ts_tablet_manager.cc:1428] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260928 21:09:16.602115 3908 ts_tablet_manager.cc:1428] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:09:16.602542 3889 heartbeater.cc:503] Master 127.21.140.126:44961 was elected leader, sending a full tablet report...
I20260928 21:09:16.604179 3907 raft_consensus.cc:348] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.604451 3907 raft_consensus.cc:374] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:16.604509 3907 raft_consensus.cc:729] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507e5ac2fbd14615aea7686c67ba5c16, State: Initialized, Role: FOLLOWER
I20260928 21:09:16.604931 3907 consensus_queue.cc:260] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.605917 3907 ts_tablet_manager.cc:1428] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:09:16.606279 3755 heartbeater.cc:503] Master 127.21.140.126:44961 was elected leader, sending a full tablet report...
W20260928 21:09:16.782362 3890 tablet.cc:2369] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:09:16.808373 3756 tablet.cc:2369] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:09:16.821326 3621 tablet.cc:2369] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:09:16.995882 3912 raft_consensus.cc:482] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:16.996110 3912 raft_consensus.cc:504] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:16.997391 3912 leader_election.cc:288] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
I20260928 21:09:17.001744 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.998123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35783 (local address 127.21.140.65:48007)
0928 21:09:16.998424 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.998436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.998524 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.998751 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.998753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.998766 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:16.998876 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.998880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.999631 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.999634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.000594 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.000603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.001277 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.001283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.001389 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.001413 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.001492 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.001550 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":21,"thread_start_us":119,"threads_started":1}
I20260928 21:09:17.001751 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.998254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:48007 (local address 127.21.140.67:35783)
0928 21:09:16.998389 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.998394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.998514 (+ 120us) server_negotiation.cc:407] Connection header received
0928 21:09:16.998597 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.998600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.998651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.998731 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:16.998981 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.999506 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.000732 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.001155 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.001253 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.001293 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.001517 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.001619 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:09:17.001855 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.997774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:52435 (local address 127.21.140.66:35281)
0928 21:09:16.997944 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:16.997949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:16.998171 (+ 222us) server_negotiation.cc:407] Connection header received
0928 21:09:16.998396 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:16.998400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:16.998463 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:16.998536 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:16.998994 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.999618 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.000846 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.001255 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.001377 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.001423 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.001647 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.001724 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:09:17.001793 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:16.997632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:35281 (local address 127.21.140.65:52435)
0928 21:09:16.998050 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:16.998066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:16.998231 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:16.998697 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:16.998701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:16.998724 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:16.998868 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:16.998875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:16.999735 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:16.999738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.000641 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.000665 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.001385 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.001388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.001499 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.001522 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.001596 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.001657 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":18,"thread_start_us":128,"threads_started":1}
I20260928 21:09:17.002354 3913 raft_consensus.cc:482] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:17.002559 3913 raft_consensus.cc:504] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:17.002691 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58db553e7fe94fa5877fe4a16658549c" candidate_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d36e1916f704be691697c9e2851f5f7" is_pre_election: true
I20260928 21:09:17.002955 3708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58db553e7fe94fa5877fe4a16658549c" candidate_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507e5ac2fbd14615aea7686c67ba5c16" is_pre_election: true
I20260928 21:09:17.003041 3843 raft_consensus.cc:2454] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6a11c5b2114197a6ad5d432b7be92b in term 0.
I20260928 21:09:17.003233 3708 raft_consensus.cc:2454] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6a11c5b2114197a6ad5d432b7be92b in term 0.
I20260928 21:09:17.003530 3506 leader_election.cc:302] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6a11c5b2114197a6ad5d432b7be92b, 5d36e1916f704be691697c9e2851f5f7; no voters:
I20260928 21:09:17.003858 3912 raft_consensus.cc:2781] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:09:17.003903 3913 leader_election.cc:288] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919)
I20260928 21:09:17.003960 3912 raft_consensus.cc:482] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:09:17.004015 3912 raft_consensus.cc:3037] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:17.005234 3912 raft_consensus.cc:504] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:17.005498 3912 leader_election.cc:288] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 1 election: Requested vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
I20260928 21:09:17.005884 3708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58db553e7fe94fa5877fe4a16658549c" candidate_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507e5ac2fbd14615aea7686c67ba5c16"
I20260928 21:09:17.005884 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58db553e7fe94fa5877fe4a16658549c" candidate_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d36e1916f704be691697c9e2851f5f7"
I20260928 21:09:17.006033 3708 raft_consensus.cc:3037] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:17.006063 3843 raft_consensus.cc:3037] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:17.007347 3843 raft_consensus.cc:2454] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6a11c5b2114197a6ad5d432b7be92b in term 1.
I20260928 21:09:17.007352 3708 raft_consensus.cc:2454] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6a11c5b2114197a6ad5d432b7be92b in term 1.
I20260928 21:09:17.007680 3506 leader_election.cc:302] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507e5ac2fbd14615aea7686c67ba5c16, 5c6a11c5b2114197a6ad5d432b7be92b; no voters:
I20260928 21:09:17.007953 3912 raft_consensus.cc:2781] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:17.008189 3912 raft_consensus.cc:686] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 LEADER]: Becoming Leader. State: Replica: 5c6a11c5b2114197a6ad5d432b7be92b, State: Running, Role: LEADER
I20260928 21:09:17.008517 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.004071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:35281 (local address 127.21.140.67:54593)
0928 21:09:17.004220 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.004242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.004317 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.004838 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.004841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.004857 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:17.004956 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.004960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.005810 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.005815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.007118 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.007128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.008047 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.008051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.008183 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.008215 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.008319 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.008396 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 21:09:17.008549 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.004319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:54593 (local address 127.21.140.66:35281)
0928 21:09:17.004619 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.004624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.004637 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:09:17.004677 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.004680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.004724 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.004806 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:17.005053 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.005545 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.007331 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.007921 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.008039 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.008086 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.008346 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.008410 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260928 21:09:17.008574 3912 consensus_queue.cc:237] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:17.008947 3708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58db553e7fe94fa5877fe4a16658549c" candidate_uuid: "5d36e1916f704be691697c9e2851f5f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507e5ac2fbd14615aea7686c67ba5c16" is_pre_election: true
I20260928 21:09:17.009121 3708 raft_consensus.cc:2379] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d36e1916f704be691697c9e2851f5f7 in current term 1: Already voted for candidate 5c6a11c5b2114197a6ad5d432b7be92b in this term.
I20260928 21:09:17.009824 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.004601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:58491 (local address 127.21.140.65:32919)
0928 21:09:17.005636 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.005642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.005655 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:09:17.005692 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.005694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.005746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.005824 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:17.006430 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.007146 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.008573 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.009183 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.009270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.009312 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.009653 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.009719 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954}
I20260928 21:09:17.009840 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.004273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:32919 (local address 127.21.140.67:58491)
0928 21:09:17.004543 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.004556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.004641 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.006123 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.006126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.006187 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:17.006300 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.006307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.007306 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.007312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.008432 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.008442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.009428 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.009432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.009529 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.009546 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.009637 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.009711 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260928 21:09:17.010519 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58db553e7fe94fa5877fe4a16658549c" candidate_uuid: "5d36e1916f704be691697c9e2851f5f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" is_pre_election: true
I20260928 21:09:17.011121 3779 leader_election.cc:302] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d36e1916f704be691697c9e2851f5f7; no voters: 507e5ac2fbd14615aea7686c67ba5c16, 5c6a11c5b2114197a6ad5d432b7be92b
I20260928 21:09:17.011564 3913 raft_consensus.cc:2726] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 21:09:17.012010 3427 catalog_manager.cc:5495] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65). New cstate: current_term: 1 leader_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } health_report { overall_health: UNKNOWN } } }
I20260928 21:09:17.037216 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.033812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.21.140.65:32919)
0928 21:09:17.033941 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.033944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.033957 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:09:17.034056 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.034058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.034115 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.034200 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:17.034427 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.034939 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.035826 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.036041 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.036120 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.036391 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:17.036843 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:09:17.036939 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.037056 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.037106 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:09:17.042917 3708 raft_consensus.cc:1261] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Refusing update from remote peer 5c6a11c5b2114197a6ad5d432b7be92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:17.042960 3843 raft_consensus.cc:1261] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Refusing update from remote peer 5c6a11c5b2114197a6ad5d432b7be92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:17.043552 3912 consensus_queue.cc:1035] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:17.043761 3918 consensus_queue.cc:1035] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:17.051127 3921 mvcc.cc:205] Tried to move back new op lower bound from 7334419484840869888 to 7334419484710158336. Current Snapshot: MvccSnapshot[applied={T|T < 7334419484840869888}]
I20260928 21:09:17.051462 3922 mvcc.cc:205] Tried to move back new op lower bound from 7334419484840869888 to 7334419484710158336. Current Snapshot: MvccSnapshot[applied={T|T < 7334419484840869888}]
I20260928 21:09:17.054076 3923 mvcc.cc:205] Tried to move back new op lower bound from 7334419484840869888 to 7334419484710158336. Current Snapshot: MvccSnapshot[applied={T|T < 7334419484840869888}]
I20260928 21:09:17.059425 3427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:17.059613 3427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:17.061993 3427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Sending DeleteTablet for 3 replicas of tablet 58db553e7fe94fa5877fe4a16658549c
I20260928 21:09:17.062862 3553 tablet_service.cc:1514] Processing DeleteTablet for tablet 58db553e7fe94fa5877fe4a16658549c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:17 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:33462
I20260928 21:09:17.062844 3823 tablet_service.cc:1514] Processing DeleteTablet for tablet 58db553e7fe94fa5877fe4a16658549c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:17 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:38564
I20260928 21:09:17.063371 3932 tablet_replica.cc:330] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: stopping tablet replica
I20260928 21:09:17.063519 3688 tablet_service.cc:1514] Processing DeleteTablet for tablet 58db553e7fe94fa5877fe4a16658549c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:17 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:38866
I20260928 21:09:17.063676 3932 raft_consensus.cc:2229] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:09:17.063690 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3490
I20260928 21:09:17.064079 3932 raft_consensus.cc:2258] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:17.064383 3931 tablet_replica.cc:330] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: stopping tablet replica
I20260928 21:09:17.064668 3931 raft_consensus.cc:2229] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:17.064949 3931 raft_consensus.cc:2258] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:17.065644 3933 tablet_replica.cc:330] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: stopping tablet replica
I20260928 21:09:17.065920 3933 raft_consensus.cc:2229] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:17.066237 3933 raft_consensus.cc:2258] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:17.066414 3931 ts_tablet_manager.cc:1905] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:17.067588 3933 ts_tablet_manager.cc:1905] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:17.069664 3931 ts_tablet_manager.cc:1918] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:17.069770 3931 log.cc:1196] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/58db553e7fe94fa5877fe4a16658549c
I20260928 21:09:17.070112 3931 ts_tablet_manager.cc:1939] T 58db553e7fe94fa5877fe4a16658549c P 5d36e1916f704be691697c9e2851f5f7: Deleting consensus metadata
I20260928 21:09:17.070556 3933 ts_tablet_manager.cc:1918] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:17.070650 3933 log.cc:1196] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/58db553e7fe94fa5877fe4a16658549c
I20260928 21:09:17.070958 3933 ts_tablet_manager.cc:1939] T 58db553e7fe94fa5877fe4a16658549c P 507e5ac2fbd14615aea7686c67ba5c16: Deleting consensus metadata
I20260928 21:09:17.071624 3411 catalog_manager.cc:4841] TS 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783): tablet 58db553e7fe94fa5877fe4a16658549c (table test-table [id=ac352edcdbf2408c903c30dfc0c84d4e]) successfully deleted
W20260928 21:09:17.071780 3414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:32919 (error 108)
I20260928 21:09:17.072062 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3625
I20260928 21:09:17.072075 3411 catalog_manager.cc:4841] TS 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281): tablet 58db553e7fe94fa5877fe4a16658549c (table test-table [id=ac352edcdbf2408c903c30dfc0c84d4e]) successfully deleted
I20260928 21:09:17.073187 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.072807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:32919 (local address 127.0.0.1:33478)
0928 21:09:17.073003 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.073081 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 21:09:17.073381 3414 catalog_manager.cc:4584] TS 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58db553e7fe94fa5877fe4a16658549c: Network error: Client connection negotiation failed: client connection to 127.21.140.65:32919: connect: Connection refused (error 111)
I20260928 21:09:17.078303 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3760
I20260928 21:09:17.083905 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3395
I20260928 21:09:17.090276 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44961
--webserver_interface=127.21.140.126
--webserver_port=43977
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:17.199344 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:17.199644 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:17.199720 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:17.203727 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:17.203814 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:17.203850 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:17.203902 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:17.203953 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:17.209286 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:17.210330 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:17.211316 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:17.224717 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:17.224717 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:17.224783 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:17.225051 3935 server_base.cc:1034] running on GCE node
I20260928 21:09:17.225700 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:17.226938 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:17.228133 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790629757228117 us; error 30 us; skew 500 ppm
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:17.230935 3935 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:17.231838 3935 webserver.cc:460] Webserver started at http://127.21.140.126:43977/ using document root <none> and password file <none>
I20260928 21:09:17.232421 3935 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:17.232520 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:17.236402 3935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:09:17.238610 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.239825 3935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:17.239970 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "8ff579ba05c445f8a2552f525dfa6467"
format_stamp: "Formatted at 2026-09-28 21:09:15 on dist-test-slave-d5wc"
server_key: "a6f4b052bf43784c1580cedaa3a99e1c"
server_key_iv: "e5a582855739658c4dfdfd22dd461503"
server_key_version: "encryptionkey@0"
I20260928 21:09:17.240584 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:17.271415 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:17.274626 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:17.274855 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:17.281723 3935 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44961
I20260928 21:09:17.282047 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44961 every 8 connection(s)
I20260928 21:09:17.282892 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:17.286608 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 3935
I20260928 21:09:17.287098 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:32919
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:44961
--webserver_port=46599
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:17.287636 4004 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:09:17.289757 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap starting.
I20260928 21:09:17.306845 4004 log.cc:824] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:17.315549 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:17.315929 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap complete.
I20260928 21:09:17.319975 4004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:17.320374 4004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff579ba05c445f8a2552f525dfa6467, State: Initialized, Role: FOLLOWER
I20260928 21:09:17.320883 4004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:17.320995 4004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:17.321086 4004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:17.321199 4004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:09:17.323163 4004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:17.323498 4004 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff579ba05c445f8a2552f525dfa6467; no voters:
I20260928 21:09:17.323822 4004 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:09:17.323927 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:09:17.324147 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 LEADER]: Becoming Leader. State: Replica: 8ff579ba05c445f8a2552f525dfa6467, State: Running, Role: LEADER
I20260928 21:09:17.324437 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:17.325065 4004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:17.325357 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff579ba05c445f8a2552f525dfa6467. Latest consensus state: current_term: 2 leader_uuid: "8ff579ba05c445f8a2552f525dfa6467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } } }
I20260928 21:09:17.325480 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:17.325846 4012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:17.326386 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8ff579ba05c445f8a2552f525dfa6467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } } }
I20260928 21:09:17.326598 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:17.330700 4012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:17.331418 4012 catalog_manager.cc:1259] Loaded cluster ID: 3170d96bd561400bb3165092940e18f7
I20260928 21:09:17.331472 4012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:17.332602 4012 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 21:09:17.335413 4012 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 21:09:17.339970 4012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:17.340644 4012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Loaded TSK: 0
I20260928 21:09:17.341672 4012 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:09:17.416651 4006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:17.416965 4006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:17.417040 4006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:17.421159 4006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:17.421248 4006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:17.421342 4006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:17.427074 4006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:32919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=46599
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:17.428179 4006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:17.429200 4006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:17.439648 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:17.439625 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:17.439867 4006 server_base.cc:1034] running on GCE node
W20260928 21:09:17.439924 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:17.440368 4006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:17.441042 4006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:17.442219 4006 hybrid_clock.cc:648] HybridClock initialized: now 1790629757442196 us; error 43 us; skew 500 ppm
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:17.445169 4006 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:17.446077 4006 webserver.cc:460] Webserver started at http://127.21.140.65:46599/ using document root <none> and password file <none>
I20260928 21:09:17.446621 4006 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:17.446722 4006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:17.450579 4006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:09:17.452919 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.454327 4006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:09:17.454443 4006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "5c6a11c5b2114197a6ad5d432b7be92b"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "b22d00604bfdd86f4eb51c9f7a9e4367"
server_key_iv: "ac8046855ab00f6195d0e57c714d8cfa"
server_key_version: "encryptionkey@0"
I20260928 21:09:17.455000 4006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:17.466770 4006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:17.470463 4006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:17.470676 4006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:17.471328 4006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:17.472442 4048 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:09:17.477782 4006 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:09:17.477877 4006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 21:09:17.478020 4006 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:09:17.480074 4006 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:09:17.480132 4006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:09:17.480635 4048 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap starting.
I20260928 21:09:17.491169 4006 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:32919
I20260928 21:09:17.492466 4006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:17.493458 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:32919 every 8 connection(s)
I20260928 21:09:17.493928 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4006
I20260928 21:09:17.494446 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:35281
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:44961
--webserver_port=40721
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:17.503623 4048 log.cc:824] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:17.511988 4048 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:17.512395 4048 tablet_bootstrap.cc:492] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap complete.
I20260928 21:09:17.513906 4048 ts_tablet_manager.cc:1397] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.005s
I20260928 21:09:17.515708 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.497481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:48395 (local address 127.21.140.126:44961)
0928 21:09:17.497998 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.498019 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.498610 (+ 591us) server_negotiation.cc:407] Connection header received
0928 21:09:17.500117 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.500136 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.500645 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.501057 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:17.502686 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.504454 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.506069 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.506793 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.506953 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.509996 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:17.510003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:17.510019 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:17.510052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:17.512092 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.512747 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.512756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.512765 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.512922 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.513303 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.513310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.513314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.513784 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:17.513934 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.514618 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.514921 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":114,"threads_started":1}
I20260928 21:09:17.515712 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.497473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:48395)
0928 21:09:17.498389 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.498455 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.499597 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.501181 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.501197 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.501791 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:17.502486 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.502505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.504618 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.504622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.505914 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.505924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.506948 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.506952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.507142 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.507836 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:17.507856 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:17.509736 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:17.512228 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.512234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.512247 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.512606 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.513024 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.513027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.513029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.513191 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.513907 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:17.513913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:17.514246 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.514447 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.514770 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":138,"threads_started":1}
I20260928 21:09:17.516783 4048 raft_consensus.cc:348] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:17.517232 4048 raft_consensus.cc:729] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6a11c5b2114197a6ad5d432b7be92b, State: Initialized, Role: FOLLOWER
I20260928 21:09:17.517699 4048 consensus_queue.cc:260] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } }
I20260928 21:09:17.518630 4157 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44961
I20260928 21:09:17.518725 4048 ts_tablet_manager.cc:1428] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260928 21:09:17.518853 4157 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:17.519366 4157 heartbeater.cc:511] Master 127.21.140.126:44961 requested a full tablet report, sending...
I20260928 21:09:17.521857 3968 ts_manager.cc:194] Registered new tserver with Master: 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919)
I20260928 21:09:17.525213 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:48395
I20260928 21:09:17.527894 4157 heartbeater.cc:503] Master 127.21.140.126:44961 was elected leader, sending a full tablet report...
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:09:17.537859 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.524552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:32919 (local address 127.0.0.1:33494)
0928 21:09:17.525289 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.525337 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.525484 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.526204 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.526211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.528469 (+ 2258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:17.528619 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.528628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.530064 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.530068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.530734 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.530744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.531597 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.531600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.531690 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.532260 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:17.532272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:17.533898 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:17.536073 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.536081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.536085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.536308 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.536625 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.536627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.536628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.536708 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.537184 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:17.537188 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:17.537321 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.537481 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.537723 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":111,"threads_started":1}
I20260928 21:09:17.537885 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.524649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.21.140.65:32919)
0928 21:09:17.525586 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.525597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.525619 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:09:17.525781 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.525787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.526021 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.526818 (+ 797us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:17.528799 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.529905 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.530851 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.531444 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.531560 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.534105 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:17.534115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:17.534128 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:17.534156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:17.535947 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.536422 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.536430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.536432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.536533 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.536794 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.536798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.536799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.537081 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:17.537209 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.537569 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.537723 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":125,"threads_started":1}
I20260928 21:09:17.539136 4091 tablet_service.cc:1514] Processing DeleteTablet for tablet 58db553e7fe94fa5877fe4a16658549c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:17 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:33494
I20260928 21:09:17.540037 4166 tablet_replica.cc:330] stopping tablet replica
I20260928 21:09:17.540318 4166 raft_consensus.cc:2229] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:17.540567 4166 raft_consensus.cc:2258] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:17.541469 4166 ts_tablet_manager.cc:1905] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:17.546726 4166 ts_tablet_manager.cc:1918] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:17.546836 4166 log.cc:1196] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/58db553e7fe94fa5877fe4a16658549c
I20260928 21:09:17.547106 4166 ts_tablet_manager.cc:1939] T 58db553e7fe94fa5877fe4a16658549c P 5c6a11c5b2114197a6ad5d432b7be92b: Deleting consensus metadata
I20260928 21:09:17.548815 3955 catalog_manager.cc:4841] TS 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919): tablet 58db553e7fe94fa5877fe4a16658549c (table test-table [id=ac352edcdbf2408c903c30dfc0c84d4e]) successfully deleted
W20260928 21:09:17.621676 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:17.621915 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:17.621958 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:17.626283 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:17.626371 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:17.626444 4159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:17.631747 4159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:35281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=40721
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:17.632702 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:17.633623 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:09:17.643567 4159 server_base.cc:1034] running on GCE node
W20260928 21:09:17.643532 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:17.643522 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:17.643533 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:17.644179 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:17.644847 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:17.646051 4159 hybrid_clock.cc:648] HybridClock initialized: now 1790629757646041 us; error 52 us; skew 500 ppm
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:17.648710 4159 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:17.649547 4159 webserver.cc:460] Webserver started at http://127.21.140.66:40721/ using document root <none> and password file <none>
I20260928 21:09:17.650111 4159 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:17.650187 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:17.654107 4159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:09:17.656183 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.657332 4159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:17.657431 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "507e5ac2fbd14615aea7686c67ba5c16"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "cc2071e5168687cf1f5ae21865e9205f"
server_key_iv: "c265f6722e11537526a82ae8e82d114a"
server_key_version: "encryptionkey@0"
I20260928 21:09:17.657945 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:17.671963 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:17.674793 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:17.674940 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:17.675436 4159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:17.676263 4159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:17.676311 4159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.676350 4159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:17.676366 4159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.685352 4159 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:35281
I20260928 21:09:17.685374 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:35281 every 8 connection(s)
I20260928 21:09:17.686390 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:17.696390 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4159
I20260928 21:09:17.696784 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:35783
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:44961
--webserver_port=33837
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:17.700259 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.688116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:32865 (local address 127.21.140.126:44961)
0928 21:09:17.688254 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.688258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.689010 (+ 752us) server_negotiation.cc:407] Connection header received
0928 21:09:17.690438 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.690442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.690505 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.690582 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:17.691707 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.692244 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.693398 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.693642 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.693756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.696249 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:17.696252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:17.696254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:17.696279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:17.697779 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.698448 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.698451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.698452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.698538 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.698937 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.698939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.698940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.699113 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:17.699188 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.699827 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.699991 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:09:17.700891 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.688444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:32865)
0928 21:09:17.688869 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.688910 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.690231 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.690755 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.690761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.691097 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:17.691548 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.691562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.692374 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.692378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.693264 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.693273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.693792 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.693796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.693988 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.694671 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:17.694693 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:17.696057 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:17.697903 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.697909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.697921 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.698333 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.698640 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.698645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.698649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.698837 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.699212 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:17.699219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:17.699519 (+ 300us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.699716 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.699960 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260928 21:09:17.702199 4296 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44961
I20260928 21:09:17.702425 4296 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:17.702956 4296 heartbeater.cc:511] Master 127.21.140.126:44961 requested a full tablet report, sending...
I20260928 21:09:17.704051 3968 ts_manager.cc:194] Registered new tserver with Master: 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281)
I20260928 21:09:17.704936 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:32865
W20260928 21:09:17.809453 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:17.809703 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:17.809754 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:17.813814 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:17.813872 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:17.813931 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:17.819190 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:35783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=33837
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:17.820132 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:17.821339 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:17.831269 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:17.831394 4299 server_base.cc:1034] running on GCE node
W20260928 21:09:17.831280 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:17.831552 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:17.831966 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:17.832581 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:17.833783 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790629757833769 us; error 28 us; skew 500 ppm
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:17.836385 4299 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:17.837188 4299 webserver.cc:460] Webserver started at http://127.21.140.67:33837/ using document root <none> and password file <none>
I20260928 21:09:17.837693 4299 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:17.837770 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:17.841490 4299 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:17.844462 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.845745 4299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:17.845875 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "5d36e1916f704be691697c9e2851f5f7"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "fc39af8b94609231057e55c162f4c17b"
server_key_iv: "ebd8483ed72eb63c7bfa747a11f984d5"
server_key_version: "encryptionkey@0"
I20260928 21:09:17.846477 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:17.859123 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:17.862246 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:17.862444 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:17.863045 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:17.863864 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:17.863930 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.863997 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:17.864042 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:17.873157 4299 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35783
I20260928 21:09:17.873174 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35783 every 8 connection(s)
I20260928 21:09:17.874229 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:09:17 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:17.883944 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4299
I20260928 21:09:17.888085 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.876022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:57093 (local address 127.21.140.126:44961)
0928 21:09:17.876174 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.876179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.877029 (+ 850us) server_negotiation.cc:407] Connection header received
0928 21:09:17.878414 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.878417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.878463 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.878541 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:17.879661 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.880194 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.881365 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.881582 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.881686 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.884486 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:17.884490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:17.884492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:17.884522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:17.886081 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.886574 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.886578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.886580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.886665 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.886994 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.886997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.886998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.887159 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:17.887242 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.887812 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.887908 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:09:17.888934 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.876445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:57093)
0928 21:09:17.876875 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.876917 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.878197 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.878698 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.878704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.879055 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:17.879527 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.879540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.880340 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.880343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.881221 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.881239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.881764 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.881768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.881944 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.882662 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:17.882679 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:17.884263 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:17.886187 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.886193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.886218 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.886484 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.886756 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:17.886758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:17.886759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:17.886885 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:17.887270 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:17.887276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:17.887565 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.887765 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.888084 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":115,"threads_started":1}
I20260928 21:09:17.890122 4429 heartbeater.cc:347] Connected to a master server at 127.21.140.126:44961
I20260928 21:09:17.890400 4429 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:17.891014 4429 heartbeater.cc:511] Master 127.21.140.126:44961 requested a full tablet report, sending...
I20260928 21:09:17.892460 3968 ts_manager.cc:194] Registered new tserver with Master: 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
I20260928 21:09:17.893142 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:57093
I20260928 21:09:17.893496 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.884631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.21.140.126:44961)
0928 21:09:17.884934 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.884939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.885102 (+ 163us) server_negotiation.cc:407] Connection header received
0928 21:09:17.885256 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.885259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.885308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.885401 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:17.886216 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.886857 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.887967 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.888295 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.888425 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.889870 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:17.889874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:17.889876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:17.889900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:17.891519 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.892070 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.892073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.892075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.892163 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:17.892616 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:17.892619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:17.892620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:17.892812 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:17.892956 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.893173 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.893305 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
I20260928 21:09:17.895211 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:09:17.899075 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.895755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.21.140.126:44961)
0928 21:09:17.895862 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.895865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.896054 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:09:17.896171 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.896173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.896254 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.896335 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:17.896543 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.897048 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.897812 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.898095 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.898171 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.898445 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:17.898798 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:09:17.898884 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.898942 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.898987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:09:17.899655 3968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:09:17.902809 3968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:09:17.910296 4091 tablet_service.cc:1467] Processing CreateTablet for tablet bdb9380debfb4e238ed70bd58936493b (DEFAULT_TABLE table=test-table [id=2fc6ad64a37246739d1cebe05ad7fec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:17.910987 4091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb9380debfb4e238ed70bd58936493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:17.913645 4048 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap starting.
I20260928 21:09:17.914604 4048 tablet_bootstrap.cc:654] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:17.914863 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.909197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:35281 (local address 127.0.0.1:38868)
0928 21:09:17.909317 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.909333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.909416 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.910274 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.910278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.910296 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:17.910386 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.910391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.912111 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.912114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.913200 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.913210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.914351 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.914355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.914459 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.914528 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.914592 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.914704 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260928 21:09:17.915114 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.909444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35783 (local address 127.0.0.1:38574)
0928 21:09:17.909801 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:17.909814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:17.909906 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:17.910622 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:17.910625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:17.910642 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:17.910726 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.910732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.911898 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.911903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.913162 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:17.913173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.914587 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.914590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:17.914803 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.914824 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:09:17.914894 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:09:17.914980 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":75,"thread_start_us":75,"threads_started":1}
I20260928 21:09:17.915670 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.909829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.21.140.67:35783)
0928 21:09:17.910182 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.910194 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.910218 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:09:17.910287 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.910292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.910479 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.910638 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:17.910816 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.911771 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.913308 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.914446 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.914792 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.915347 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.915404 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.915517 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":186,"threads_started":1}
I20260928 21:09:17.915689 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:17.909369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.21.140.66:35281)
0928 21:09:17.909730 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:17.909743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:17.909767 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:09:17.909843 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:17.909850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:17.910104 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:17.910263 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:17.910604 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.911977 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.913338 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:17.914206 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:17.915173 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:17.915425 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:17.915484 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 21:09:17.915575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":141,"threads_started":1}
I20260928 21:09:17.916174 4048 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: No bootstrap required, opened a new log
I20260928 21:09:17.916285 4048 ts_tablet_manager.cc:1397] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:17.916769 4048 raft_consensus.cc:348] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:17.916896 4048 raft_consensus.cc:374] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:17.916937 4048 raft_consensus.cc:729] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6a11c5b2114197a6ad5d432b7be92b, State: Initialized, Role: FOLLOWER
I20260928 21:09:17.917075 4048 consensus_queue.cc:260] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:17.917366 4048 ts_tablet_manager.cc:1428] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:09:17.918895 4230 tablet_service.cc:1467] Processing CreateTablet for tablet bdb9380debfb4e238ed70bd58936493b (DEFAULT_TABLE table=test-table [id=2fc6ad64a37246739d1cebe05ad7fec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:17.919845 4230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb9380debfb4e238ed70bd58936493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:17.920215 4363 tablet_service.cc:1467] Processing CreateTablet for tablet bdb9380debfb4e238ed70bd58936493b (DEFAULT_TABLE table=test-table [id=2fc6ad64a37246739d1cebe05ad7fec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:17.921103 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb9380debfb4e238ed70bd58936493b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:17.926821 4443 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Bootstrap starting.
I20260928 21:09:17.929306 4443 tablet_bootstrap.cc:654] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:17.930008 4443 log.cc:824] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:17.930320 4444 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Bootstrap starting.
I20260928 21:09:17.931701 4443 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: No bootstrap required, opened a new log
I20260928 21:09:17.931893 4443 ts_tablet_manager.cc:1397] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:09:17.932067 4444 tablet_bootstrap.cc:654] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:17.933321 4444 log.cc:824] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:17.934984 4444 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: No bootstrap required, opened a new log
I20260928 21:09:17.935189 4444 ts_tablet_manager.cc:1397] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:09:17.935387 4443 raft_consensus.cc:348] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:17.935596 4443 raft_consensus.cc:374] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:17.935670 4443 raft_consensus.cc:729] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507e5ac2fbd14615aea7686c67ba5c16, State: Initialized, Role: FOLLOWER
I20260928 21:09:17.936075 4443 consensus_queue.cc:260] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:17.936710 4296 heartbeater.cc:503] Master 127.21.140.126:44961 was elected leader, sending a full tablet report...
I20260928 21:09:17.936982 4443 ts_tablet_manager.cc:1428] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 21:09:17.937259 4297 tablet.cc:2369] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:09:17.938710 4444 raft_consensus.cc:348] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:17.938956 4444 raft_consensus.cc:374] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:17.939011 4444 raft_consensus.cc:729] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d36e1916f704be691697c9e2851f5f7, State: Initialized, Role: FOLLOWER
I20260928 21:09:17.939424 4444 consensus_queue.cc:260] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:17.940092 4429 heartbeater.cc:503] Master 127.21.140.126:44961 was elected leader, sending a full tablet report...
I20260928 21:09:17.940429 4444 ts_tablet_manager.cc:1428] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260928 21:09:17.994362 4158 tablet.cc:2369] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:09:18.125178 4430 tablet.cc:2369] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:09:18.293581 4448 raft_consensus.cc:482] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:18.293823 4448 raft_consensus.cc:504] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:18.295053 4448 leader_election.cc:288] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919)
I20260928 21:09:18.298802 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.295381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:35281 (local address 127.21.140.67:41289)
0928 21:09:18.295515 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:18.295532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:18.295636 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:18.295978 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:18.295981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:18.296008 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:18.296126 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:18.296131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.296870 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.296876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:18.297632 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:18.297641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.298357 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.298361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:18.298476 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:18.298494 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:09:18.298577 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:09:18.298656 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"spinlock_wait_cycles":1280}
I20260928 21:09:18.298802 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.295495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:41289 (local address 127.21.140.66:35281)
0928 21:09:18.295621 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:18.295626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:18.295645 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:09:18.295826 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:18.295829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:18.295876 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:18.295965 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:18.296263 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.296753 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.297770 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.298233 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.298362 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:18.298420 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:18.298609 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:09:18.298679 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:09:18.299289 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.295583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:33895 (local address 127.21.140.65:32919)
0928 21:09:18.295873 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:18.295878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:18.295961 (+ 83us) server_negotiation.cc:407] Connection header received
0928 21:09:18.296117 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:18.296120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:18.296181 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:18.296295 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:18.296561 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.297060 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.298121 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.298565 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.298708 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:18.298867 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:18.299065 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 21:09:18.299149 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
I20260928 21:09:18.299288 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.295381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:32919 (local address 127.21.140.67:33895)
0928 21:09:18.295850 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:18.295865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:18.295998 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:18.296303 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:18.296306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:18.296321 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:18.296446 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:18.296452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.297178 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.297183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:18.297988 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:18.297997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.298696 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.298700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:18.298790 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:18.298811 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:09:18.299032 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 21:09:18.299107 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":140,"thread_start_us":105,"threads_started":1}
I20260928 21:09:18.299548 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb9380debfb4e238ed70bd58936493b" candidate_uuid: "5d36e1916f704be691697c9e2851f5f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507e5ac2fbd14615aea7686c67ba5c16" is_pre_election: true
I20260928 21:09:18.299885 4250 raft_consensus.cc:2454] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d36e1916f704be691697c9e2851f5f7 in term 0.
I20260928 21:09:18.299870 4111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb9380debfb4e238ed70bd58936493b" candidate_uuid: "5d36e1916f704be691697c9e2851f5f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" is_pre_election: true
I20260928 21:09:18.300143 4111 raft_consensus.cc:2454] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d36e1916f704be691697c9e2851f5f7 in term 0.
I20260928 21:09:18.300371 4316 leader_election.cc:302] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507e5ac2fbd14615aea7686c67ba5c16, 5d36e1916f704be691697c9e2851f5f7; no voters:
I20260928 21:09:18.300693 4448 raft_consensus.cc:2781] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:09:18.300774 4448 raft_consensus.cc:482] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:09:18.300827 4448 raft_consensus.cc:3037] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:18.302012 4448 raft_consensus.cc:504] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:18.302397 4448 leader_election.cc:288] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [CANDIDATE]: Term 1 election: Requested vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919)
I20260928 21:09:18.302791 4111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb9380debfb4e238ed70bd58936493b" candidate_uuid: "5d36e1916f704be691697c9e2851f5f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6a11c5b2114197a6ad5d432b7be92b"
I20260928 21:09:18.302922 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb9380debfb4e238ed70bd58936493b" candidate_uuid: "5d36e1916f704be691697c9e2851f5f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507e5ac2fbd14615aea7686c67ba5c16"
I20260928 21:09:18.302934 4111 raft_consensus.cc:3037] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:18.303073 4250 raft_consensus.cc:3037] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:18.303997 4111 raft_consensus.cc:2454] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d36e1916f704be691697c9e2851f5f7 in term 1.
I20260928 21:09:18.304087 4250 raft_consensus.cc:2454] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d36e1916f704be691697c9e2851f5f7 in term 1.
I20260928 21:09:18.304342 4319 leader_election.cc:302] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6a11c5b2114197a6ad5d432b7be92b, 5d36e1916f704be691697c9e2851f5f7; no voters:
I20260928 21:09:18.304550 4448 raft_consensus.cc:2781] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:18.304771 4448 raft_consensus.cc:686] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5d36e1916f704be691697c9e2851f5f7, State: Running, Role: LEADER
I20260928 21:09:18.305121 4448 consensus_queue.cc:237] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:18.308308 3968 catalog_manager.cc:5495] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "5d36e1916f704be691697c9e2851f5f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } health_report { overall_health: UNKNOWN } } }
I20260928 21:09:18.368559 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.364919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.21.140.67:35783)
0928 21:09:18.365057 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:18.365061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:18.365076 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:09:18.365167 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:18.365169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:18.365224 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:18.365308 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:18.365537 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.366104 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.366936 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:18.367154 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:18.367241 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:18.367562 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:18.368041 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:09:18.368247 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:18.368312 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:09:18.368370 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:09:18.374540 4250 raft_consensus.cc:1261] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Refusing update from remote peer 5d36e1916f704be691697c9e2851f5f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:18.374537 4111 raft_consensus.cc:1261] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Refusing update from remote peer 5d36e1916f704be691697c9e2851f5f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:18.375270 4448 consensus_queue.cc:1035] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:18.375486 4451 consensus_queue.cc:1035] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:18.381278 4454 mvcc.cc:205] Tried to move back new op lower bound from 7334419490294550528 to 7334419490020515840. Current Snapshot: MvccSnapshot[applied={T|T < 7334419490294550528}]
I20260928 21:09:18.388820 4456 mvcc.cc:205] Tried to move back new op lower bound from 7334419490294550528 to 7334419490020515840. Current Snapshot: MvccSnapshot[applied={T|T < 7334419490294550528 or (T in {7334419490294550528})}]
I20260928 21:09:18.389885 3968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:18.390092 3968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:18.392495 3968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Sending DeleteTablet for 3 replicas of tablet bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:18.393298 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb9380debfb4e238ed70bd58936493b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:18 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:38574
I20260928 21:09:18.393321 4230 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb9380debfb4e238ed70bd58936493b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:18 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:38868
I20260928 21:09:18.393802 4463 tablet_replica.cc:330] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: stopping tablet replica
I20260928 21:09:18.393775 4091 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb9380debfb4e238ed70bd58936493b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:18 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:33494
I20260928 21:09:18.393846 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 4006
I20260928 21:09:18.394230 4463 raft_consensus.cc:2229] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:09:18.394611 4463 raft_consensus.cc:2258] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:18.394757 4464 tablet_replica.cc:330] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: stopping tablet replica
I20260928 21:09:18.395045 4464 raft_consensus.cc:2229] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:18.395367 4464 raft_consensus.cc:2258] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:18.396413 4463 ts_tablet_manager.cc:1905] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:18.396656 4464 ts_tablet_manager.cc:1905] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:18.399266 4463 ts_tablet_manager.cc:1918] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:18.399430 4463 log.cc:1196] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:18.399457 4464 ts_tablet_manager.cc:1918] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:18.399545 4464 log.cc:1196] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:18.399904 4463 ts_tablet_manager.cc:1939] T bdb9380debfb4e238ed70bd58936493b P 5d36e1916f704be691697c9e2851f5f7: Deleting consensus metadata
I20260928 21:09:18.399904 4464 ts_tablet_manager.cc:1939] T bdb9380debfb4e238ed70bd58936493b P 507e5ac2fbd14615aea7686c67ba5c16: Deleting consensus metadata
I20260928 21:09:18.401080 3952 catalog_manager.cc:4841] TS 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281): tablet bdb9380debfb4e238ed70bd58936493b (table test-table [id=2fc6ad64a37246739d1cebe05ad7fec4]) successfully deleted
I20260928 21:09:18.401290 3952 catalog_manager.cc:4841] TS 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783): tablet bdb9380debfb4e238ed70bd58936493b (table test-table [id=2fc6ad64a37246739d1cebe05ad7fec4]) successfully deleted
W20260928 21:09:18.404243 3955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:32919 (error 108)
I20260928 21:09:18.404778 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 4159
I20260928 21:09:18.405315 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.404964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:32919 (local address 127.0.0.1:33504)
0928 21:09:18.405151 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:18.405219 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:32919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 21:09:18.405489 3955 catalog_manager.cc:4584] TS 5c6a11c5b2114197a6ad5d432b7be92b (127.21.140.65:32919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdb9380debfb4e238ed70bd58936493b: Network error: Client connection negotiation failed: client connection to 127.21.140.65:32919: connect: Connection refused (error 111)
I20260928 21:09:18.411134 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 4299
I20260928 21:09:18.417382 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 3935
I20260928 21:09:18.423651 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44961
--webserver_interface=127.21.140.126
--webserver_port=43977
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:18.535939 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:18.536268 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:18.536343 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:18.540407 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:18.540504 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:18.540541 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:18.540611 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:18.540652 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:18.545997 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:18.547086 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:18.548139 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:18.556559 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:18.556605 4466 server_base.cc:1034] running on GCE node
W20260928 21:09:18.556566 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:18.556571 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:18.557351 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:18.558516 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:18.559716 4466 hybrid_clock.cc:648] HybridClock initialized: now 1790629758559695 us; error 31 us; skew 500 ppm
Sep 28 21:09:18 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629758, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:18.562393 4466 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:18.563184 4466 webserver.cc:460] Webserver started at http://127.21.140.126:43977/ using document root <none> and password file <none>
I20260928 21:09:18.563683 4466 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:18.563745 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:18.567428 4466 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:18.569523 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:18.570809 4466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:18.570957 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "8ff579ba05c445f8a2552f525dfa6467"
format_stamp: "Formatted at 2026-09-28 21:09:15 on dist-test-slave-d5wc"
server_key: "a6f4b052bf43784c1580cedaa3a99e1c"
server_key_iv: "e5a582855739658c4dfdfd22dd461503"
server_key_version: "encryptionkey@0"
I20260928 21:09:18.571518 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:18.589006 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:18.591951 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:18.592165 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:18.599457 4466 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44961
I20260928 21:09:18.599773 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44961 every 8 connection(s)
I20260928 21:09:18.600633 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:18.604851 4534 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:09:18.606551 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap starting.
I20260928 21:09:18.610056 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4466
I20260928 21:09:18.610481 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:32919
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:44961
--webserver_port=46599
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:18.623977 4534 log.cc:824] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:18.635380 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:18.635784 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467: Bootstrap complete.
I20260928 21:09:18.639796 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:18.640205 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff579ba05c445f8a2552f525dfa6467, State: Initialized, Role: FOLLOWER
I20260928 21:09:18.640691 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:18.640878 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:18.640959 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:18.641075 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:09:18.643015 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:18.643347 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff579ba05c445f8a2552f525dfa6467; no voters:
I20260928 21:09:18.643689 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:09:18.643774 4538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:09:18.644030 4538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [term 3 LEADER]: Becoming Leader. State: Replica: 8ff579ba05c445f8a2552f525dfa6467, State: Running, Role: LEADER
I20260928 21:09:18.644338 4538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } }
I20260928 21:09:18.644968 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:18.645279 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff579ba05c445f8a2552f525dfa6467. Latest consensus state: current_term: 3 leader_uuid: "8ff579ba05c445f8a2552f525dfa6467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } } }
I20260928 21:09:18.645403 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:18.645767 4542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:18.645789 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8ff579ba05c445f8a2552f525dfa6467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff579ba05c445f8a2552f525dfa6467" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44961 } } }
I20260928 21:09:18.646060 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff579ba05c445f8a2552f525dfa6467 [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:18.651047 4542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:18.651779 4542 catalog_manager.cc:1259] Loaded cluster ID: 3170d96bd561400bb3165092940e18f7
I20260928 21:09:18.651858 4542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 21:09:18.652786 4542 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f20f333d9ad google::LogMessage::Fail() at ??:0
@ 0x7f20f334088a google::LogMessage::SendToLog() at ??:0
@ 0x7f20f333d4de google::LogMessage::Flush() at ??:0
@ 0x7f20f333ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f20fdc7cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f20fdc7dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f20fdc7a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f20fdcbe640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f20f558f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f20f558fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f20f55913f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f20f5580512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f20f6bc46db start_thread at ??:0
@ 0x7f20f1e3b61f clone at ??:0
W20260928 21:09:18.737038 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:18.737319 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:18.737406 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:18.741581 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:18.741683 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:18.741772 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:18.747534 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:32919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=46599
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:18.748653 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:18.749712 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:18.759706 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:18.759739 4536 server_base.cc:1034] running on GCE node
W20260928 21:09:18.760000 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:18.760067 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:18.760494 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:18.761224 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:18.762418 4536 hybrid_clock.cc:648] HybridClock initialized: now 1790629758762390 us; error 51 us; skew 500 ppm
Sep 28 21:09:18 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629758, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:18.765378 4536 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:18.766410 4536 webserver.cc:460] Webserver started at http://127.21.140.65:46599/ using document root <none> and password file <none>
I20260928 21:09:18.766999 4536 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:18.767062 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:18.771204 4536 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:09:18.773535 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:18.774931 4536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:18.775171 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "5c6a11c5b2114197a6ad5d432b7be92b"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "b22d00604bfdd86f4eb51c9f7a9e4367"
server_key_iv: "ac8046855ab00f6195d0e57c714d8cfa"
server_key_version: "encryptionkey@0"
I20260928 21:09:18.775750 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:18.787497 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:18.790705 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:18.790906 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:18.791482 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:18.792563 4592 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:09:18.798373 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:09:18.798460 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 21:09:18.798542 4536 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:09:18.800172 4536 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:09:18.800246 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:09:18.800643 4592 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap starting.
I20260928 21:09:18.810985 4536 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:32919
I20260928 21:09:18.811211 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:32919 every 8 connection(s)
I20260928 21:09:18.811990 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:18.816977 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4536
I20260928 21:09:18.817413 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:35281
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:44961
--webserver_port=40721
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:18.823617 4592 log.cc:824] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:18.833802 4592 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:18.834409 4592 tablet_bootstrap.cc:492] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Bootstrap complete.
I20260928 21:09:18.836000 4592 ts_tablet_manager.cc:1397] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.014s
I20260928 21:09:18.839327 4592 raft_consensus.cc:348] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:18.839907 4592 raft_consensus.cc:729] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6a11c5b2114197a6ad5d432b7be92b, State: Initialized, Role: FOLLOWER
I20260928 21:09:18.840482 4592 consensus_queue.cc:260] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:18.841540 4592 ts_tablet_manager.cc:1428] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260928 21:09:18.908313 4703 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:09:18.813884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:38791)
0928 21:09:18.814447 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:18.814501 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:18.815335 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:18.907497 (+ 92162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":154,"threads_started":1}
W20260928 21:09:18.908823 4701 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:44961 (0 consecutive failures): Network error: Failed to ping master at 127.21.140.126:44961: Client connection negotiation failed: client connection to 127.21.140.126:44961: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:09:18.909569 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.909325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:57801)
0928 21:09:18.909450 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:18.909514 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 21:09:18.910319 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:18.910129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:39313)
0928 21:09:18.910230 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:18.910273 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:09:18.910524 4701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:09:18.934134 4704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:18.934392 4704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:18.934463 4704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:18.938522 4704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:18.938585 4704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:18.938643 4704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:18.943930 4704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:35281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=40721
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:18.944963 4704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:18.945946 4704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:18.955685 4712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:18.955760 4704 server_base.cc:1034] running on GCE node
W20260928 21:09:18.955689 4714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:18.956151 4711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:18.956552 4704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:18.957235 4704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:18.958421 4704 hybrid_clock.cc:648] HybridClock initialized: now 1790629758958403 us; error 35 us; skew 500 ppm
Sep 28 21:09:18 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629758, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:18.961037 4704 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:18.961841 4704 webserver.cc:460] Webserver started at http://127.21.140.66:40721/ using document root <none> and password file <none>
I20260928 21:09:18.962412 4704 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:18.962472 4704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:18.966073 4704 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 21:09:18.967988 4721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:18.969143 4704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:18.969275 4704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "507e5ac2fbd14615aea7686c67ba5c16"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "cc2071e5168687cf1f5ae21865e9205f"
server_key_iv: "c265f6722e11537526a82ae8e82d114a"
server_key_version: "encryptionkey@0"
I20260928 21:09:18.969795 4704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:18.984299 4704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:18.987154 4704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:18.987358 4704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:18.987936 4704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:18.988747 4704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:18.988794 4704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:18.988858 4704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:18.988893 4704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:18.997820 4704 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:35281
I20260928 21:09:18.997848 4835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:35281 every 8 connection(s)
I20260928 21:09:18.998862 4704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:09:19.002717 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.000934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:59407)
0928 21:09:19.001380 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.001493 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
W20260928 21:09:19.003142 4836 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:44961 (0 consecutive failures): Network error: Failed to ping master at 127.21.140.126:44961: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
I20260928 21:09:19.003760 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.003542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:44957)
0928 21:09:19.003650 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.003704 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 21:09:19.004328 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.004121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:46117)
0928 21:09:19.004231 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.004279 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 21:09:19.004510 4836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:09:19.005332 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4704
I20260928 21:09:19.005725 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:35783
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:44961
--webserver_port=33837
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:19.115259 4839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:19.115548 4839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:19.115625 4839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:19.119611 4839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:19.119691 4839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:19.119776 4839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:19.124943 4839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:35783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=33837
--tserver_master_addrs=127.21.140.126:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:19.125957 4839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:19.126972 4839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:19.136415 4844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:19.136409 4847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:19.136452 4845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:19.136633 4839 server_base.cc:1034] running on GCE node
I20260928 21:09:19.137080 4839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:19.137694 4839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:19.138865 4839 hybrid_clock.cc:648] HybridClock initialized: now 1790629759138831 us; error 59 us; skew 500 ppm
Sep 28 21:09:19 dist-test-slave-d5wc krb5kdc[3382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629759, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:19.141552 4839 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:19.142477 4839 webserver.cc:460] Webserver started at http://127.21.140.67:33837/ using document root <none> and password file <none>
I20260928 21:09:19.143013 4839 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:19.143100 4839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:19.146816 4839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:19.148804 4854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:19.149951 4839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:19.150125 4839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "5d36e1916f704be691697c9e2851f5f7"
format_stamp: "Formatted at 2026-09-28 21:09:16 on dist-test-slave-d5wc"
server_key: "fc39af8b94609231057e55c162f4c17b"
server_key_iv: "ebd8483ed72eb63c7bfa747a11f984d5"
server_key_version: "encryptionkey@0"
I20260928 21:09:19.150674 4839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:19.173758 4839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:19.177206 4839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:19.177414 4839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:19.178035 4839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:19.178838 4839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:19.178903 4839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:19.178989 4839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:19.179036 4839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:19.188130 4839 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35783
I20260928 21:09:19.188162 4968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35783 every 8 connection(s)
I20260928 21:09:19.189064 4839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:09:19.191964 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 4839
I20260928 21:09:19.193466 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.191386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:33223)
0928 21:09:19.191899 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.192062 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
W20260928 21:09:19.193878 4969 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:44961 (0 consecutive failures): Network error: Failed to ping master at 127.21.140.126:44961: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
I20260928 21:09:19.194481 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.194274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:51945)
0928 21:09:19.194380 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.194431 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 21:09:19.195099 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.194888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:60077)
0928 21:09:19.195011 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.195057 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 21:09:19.195252 4969 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:09:19.911578 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:19.911136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:44789)
0928 21:09:19.911394 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:19.911461 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260928 21:09:20.005818 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.005228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:50867)
0928 21:09:20.005640 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:20.005706 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260928 21:09:20.196156 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.195781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:47055)
0928 21:09:20.196006 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:20.196059 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260928 21:09:20.605664 4976 raft_consensus.cc:482] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:20.605935 4976 raft_consensus.cc:504] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:20.607554 4976 leader_election.cc:288] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
Sep 28 21:09:20 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629758, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.66@KRBTEST.COM
Sep 28 21:09:20 dist-test-slave-d5wc krb5kdc[3382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629758, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.67@KRBTEST.COM
I20260928 21:09:20.625530 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.607899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:35281 (local address 127.21.140.65:42287)
0928 21:09:20.608632 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:20.608649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:20.608794 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:20.611196 (+ 2402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:20.611203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:20.611739 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:20.612406 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:20.612429 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.614046 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.614049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:20.615280 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:20.615296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.616206 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.616211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:20.616524 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:20.617423 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:20.617445 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:20.619660 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:20.622451 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:20.622463 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:20.622487 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:20.622917 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:20.623344 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:20.623347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:20.623349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:20.623542 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:20.624310 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:20.624319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:20.624634 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 21:09:20.624850 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 21:09:20.625304 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":239,"spinlock_wait_cycles":22016,"thread_start_us":131,"threads_started":1}
I20260928 21:09:20.625547 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.608107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:42287 (local address 127.21.140.66:35281)
0928 21:09:20.608713 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:20.608737 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:20.608802 (+ 65us) server_negotiation.cc:407] Connection header received
0928 21:09:20.609929 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:20.609941 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:20.610594 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:20.611023 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:20.612650 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.613802 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.615455 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.616039 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.616267 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:20.620048 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:20.620060 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:20.620086 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:20.620129 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:20.622274 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:20.623072 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:20.623079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:20.623084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:20.623237 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:20.623647 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:20.623651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:20.623653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:20.624186 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:20.624345 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:20.624975 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 21:09:20.625292 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260928 21:09:20.625548 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.608880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:49801 (local address 127.21.140.67:35783)
0928 21:09:20.609291 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:20.609319 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:20.609363 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:09:20.610340 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:20.610349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:20.610683 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:20.611023 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:20.612651 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.613803 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.615456 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.616041 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.616266 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:20.620171 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:20.620177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:20.620189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:20.620229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:20.622335 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:20.623074 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:20.623080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:20.623085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:20.623237 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:20.623659 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:20.623662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:20.623664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:20.624188 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:20.624345 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:20.624967 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 21:09:20.625287 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":155,"threads_started":1}
I20260928 21:09:20.625583 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.608386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35783 (local address 127.21.140.65:49801)
0928 21:09:20.608808 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:20.608822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:20.608948 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:20.611631 (+ 2683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:20.611634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:20.611842 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:20.612406 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:20.612429 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.614045 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.614049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:20.615280 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:20.615296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:20.616206 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:20.616211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:20.616523 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:20.617438 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:20.617450 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:20.619929 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:20.622503 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:20.622505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:20.622507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:20.622911 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:20.623351 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:20.623354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:20.623356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:20.623563 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:20.624311 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:20.624319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:20.624635 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 21:09:20.624850 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 21:09:20.625397 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":415,"spinlock_wait_cycles":30080,"thread_start_us":77,"threads_started":1}
W20260928 21:09:20.629329 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
W20260928 21:09:20.629602 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:20.629657 4586 leader_election.cc:302] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c6a11c5b2114197a6ad5d432b7be92b; no voters: 507e5ac2fbd14615aea7686c67ba5c16, 5d36e1916f704be691697c9e2851f5f7
I20260928 21:09:20.629997 4976 raft_consensus.cc:2726] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:09:20.913259 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:20.912752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:35367)
0928 21:09:20.913034 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:20.913135 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 21:09:21.007463 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:21.006832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:45185)
0928 21:09:21.007199 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:21.007277 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
I20260928 21:09:21.197510 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:21.196976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:42291)
0928 21:09:21.197291 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:21.197378 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
I20260928 21:09:21.914819 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:21.914277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:46019)
0928 21:09:21.914587 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:21.914695 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20260928 21:09:22.008872 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:22.008373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:47363)
0928 21:09:22.008706 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:22.008767 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
I20260928 21:09:22.198808 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:22.198383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:43855)
0928 21:09:22.198640 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:22.198704 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260928 21:09:22.471473 4986 raft_consensus.cc:482] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:22.471650 4986 raft_consensus.cc:504] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:22.472121 4986 leader_election.cc:288] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
W20260928 21:09:22.473165 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
W20260928 21:09:22.473331 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:22.473395 4586 leader_election.cc:302] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c6a11c5b2114197a6ad5d432b7be92b; no voters: 507e5ac2fbd14615aea7686c67ba5c16, 5d36e1916f704be691697c9e2851f5f7
I20260928 21:09:22.473584 4986 raft_consensus.cc:2726] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:09:22.916152 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:22.915676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:60737)
0928 21:09:22.915988 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:22.916056 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
I20260928 21:09:23.010174 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:23.009706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:54303)
0928 21:09:23.009951 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:23.010056 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260928 21:09:23.199931 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:23.199516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:56057)
0928 21:09:23.199789 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:23.199843 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260928 21:09:23.918126 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:23.917438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:60671)
0928 21:09:23.917836 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:23.917922 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":107,"threads_started":1}
I20260928 21:09:24.011976 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:24.011361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:60359)
0928 21:09:24.011748 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:24.011834 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260928 21:09:24.201529 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:24.201002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:40289)
0928 21:09:24.201324 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:24.201399 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260928 21:09:24.919363 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:24.918937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:52037)
0928 21:09:24.919205 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:24.919264 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20260928 21:09:25.013545 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:25.013042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:32933)
0928 21:09:25.013378 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:25.013440 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":126,"threads_started":1}
I20260928 21:09:25.202898 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:25.202473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:38097)
0928 21:09:25.202741 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:25.202797 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260928 21:09:25.543447 4996 raft_consensus.cc:482] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:25.543720 4996 raft_consensus.cc:504] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:25.544152 4996 leader_election.cc:288] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
W20260928 21:09:25.545032 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
W20260928 21:09:25.545195 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:25.545264 4586 leader_election.cc:302] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c6a11c5b2114197a6ad5d432b7be92b; no voters: 507e5ac2fbd14615aea7686c67ba5c16, 5d36e1916f704be691697c9e2851f5f7
I20260928 21:09:25.545440 4996 raft_consensus.cc:2726] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:09:25.920802 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:25.920339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:54747)
0928 21:09:25.920636 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:25.920704 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260928 21:09:26.015152 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:26.014585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:46469)
0928 21:09:26.014970 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:26.015034 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":78,"threads_started":1}
I20260928 21:09:26.204259 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:26.203772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:49789)
0928 21:09:26.204093 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:26.204158 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":77,"threads_started":1}
I20260928 21:09:26.922591 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:26.921951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:47793)
0928 21:09:26.922386 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:26.922463 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260928 21:09:27.017019 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:27.016402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:57459)
0928 21:09:27.016796 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:27.016879 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260928 21:09:27.206137 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:27.205480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:47981)
0928 21:09:27.205850 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:27.205948 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260928 21:09:27.924365 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:27.923773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:34095)
0928 21:09:27.924124 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:27.924226 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260928 21:09:28.019074 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:28.018456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:48961)
0928 21:09:28.018837 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:28.018926 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260928 21:09:28.208016 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:28.207320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:60323)
0928 21:09:28.207761 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:28.207850 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260928 21:09:28.926299 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:28.925612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:56813)
0928 21:09:28.926064 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:28.926163 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260928 21:09:29.021000 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:29.020368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:37557)
0928 21:09:29.020766 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:29.020848 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":149,"threads_started":1}
I20260928 21:09:29.209772 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:29.209131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:55337)
0928 21:09:29.209560 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:29.209641 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260928 21:09:29.928090 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:29.927495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:50045)
0928 21:09:29.927878 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:29.927961 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260928 21:09:30.022825 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:30.022227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:58849)
0928 21:09:30.022604 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:30.022693 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260928 21:09:30.168220 5011 raft_consensus.cc:482] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:30.168437 5011 raft_consensus.cc:504] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507e5ac2fbd14615aea7686c67ba5c16" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 35281 } } peers { permanent_uuid: "5d36e1916f704be691697c9e2851f5f7" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35783 } } peers { permanent_uuid: "5c6a11c5b2114197a6ad5d432b7be92b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 32919 } }
I20260928 21:09:30.169137 5011 leader_election.cc:288] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281), 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783)
W20260928 21:09:30.170188 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d36e1916f704be691697c9e2851f5f7 (127.21.140.67:35783): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
W20260928 21:09:30.170323 4586 leader_election.cc:341] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 507e5ac2fbd14615aea7686c67ba5c16 (127.21.140.66:35281): Not found: Tablet not found: bdb9380debfb4e238ed70bd58936493b
I20260928 21:09:30.170363 4586 leader_election.cc:302] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c6a11c5b2114197a6ad5d432b7be92b; no voters: 507e5ac2fbd14615aea7686c67ba5c16, 5d36e1916f704be691697c9e2851f5f7
I20260928 21:09:30.170529 5011 raft_consensus.cc:2726] T bdb9380debfb4e238ed70bd58936493b P 5c6a11c5b2114197a6ad5d432b7be92b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:09:30.211742 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:30.211095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:59727)
0928 21:09:30.211506 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:30.211597 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260928 21:09:30.929862 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:30.929242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:47749)
0928 21:09:30.929638 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:30.929723 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":145,"threads_started":1}
I20260928 21:09:31.024420 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:31.023847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:38447)
0928 21:09:31.024199 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:31.024276 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260928 21:09:31.213675 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:31.213029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:59697)
0928 21:09:31.213433 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:31.213520 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
I20260928 21:09:31.931311 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:31.930845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:60655)
0928 21:09:31.931148 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:31.931210 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260928 21:09:32.025853 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:32.025388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:34427)
0928 21:09:32.025686 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:32.025749 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260928 21:09:32.215153 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:32.214694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:40673)
0928 21:09:32.214972 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:32.215041 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260928 21:09:32.932962 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:32.932430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:60649)
0928 21:09:32.932745 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:32.932839 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260928 21:09:33.027508 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:33.026887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:53269)
0928 21:09:33.027282 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:33.027371 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260928 21:09:33.216908 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:33.216269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.67:33635)
0928 21:09:33.216677 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:33.216759 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
I20260928 21:09:33.934749 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:33.934233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.65:55941)
0928 21:09:33.934549 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:33.934614 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260928 21:09:34.029487 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:34.028916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44961 (local address 127.21.140.66:37675)
0928 21:09:34.029275 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:34.029355 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260928 21:09:34.194317 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 4536
I20260928 21:09:34.201777 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 4704
I20260928 21:09:34.209043 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 4839
2026-09-28T21:09:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19764 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:09:34 dist-test-slave-d5wc krb5kdc[5026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:09:34 dist-test-slave-d5wc krb5kdc[5026](info): set up 2 sockets
Sep 28 21:09:34 dist-test-slave-d5wc krb5kdc[5026](info): commencing operation
krb5kdc: starting...
W20260928 21:09:35.292177 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:09:35 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:09:35.421353 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40549
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:35.533509 5039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:35.533836 5039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:35.533919 5039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:35.538019 5039 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:09:35.538112 5039 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:09:35.538142 5039 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:09:35.538178 5039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:35.538220 5039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:35.538259 5039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:35.538292 5039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:35.538326 5039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:35.543637 5039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:35.544709 5039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:35.545756 5039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:35.554579 5045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:35.554591 5044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:35.554630 5039 server_base.cc:1034] running on GCE node
W20260928 21:09:35.554591 5047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:35.555439 5039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:35.556627 5039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:35.557874 5039 hybrid_clock.cc:648] HybridClock initialized: now 1790629775557849 us; error 43 us; skew 500 ppm
Sep 28 21:09:35 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629775, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:35.561149 5039 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:35.562107 5039 webserver.cc:460] Webserver started at http://127.21.140.126:41441/ using document root <none> and password file <none>
I20260928 21:09:35.562691 5039 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:35.562754 5039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:35.562983 5039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:35.565024 5039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "1506ac661a2e463dac392a2c6c36478b"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "ace067dad802d98f5ec8fdce5b776334"
server_key_iv: "f2454b64c1a1768b1937a209f42a776d"
server_key_version: "encryptionkey@0"
I20260928 21:09:35.565609 5039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "1506ac661a2e463dac392a2c6c36478b"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "ace067dad802d98f5ec8fdce5b776334"
server_key_iv: "f2454b64c1a1768b1937a209f42a776d"
server_key_version: "encryptionkey@0"
I20260928 21:09:35.569380 5039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 21:09:35.571799 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:35.573035 5039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:09:35.573173 5039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "1506ac661a2e463dac392a2c6c36478b"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "ace067dad802d98f5ec8fdce5b776334"
server_key_iv: "f2454b64c1a1768b1937a209f42a776d"
server_key_version: "encryptionkey@0"
I20260928 21:09:35.573284 5039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:35.584612 5039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:35.587466 5039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:35.587667 5039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:35.594630 5039 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40549
I20260928 21:09:35.594940 5107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40549 every 8 connection(s)
I20260928 21:09:35.595592 5039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:35.597723 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5039
I20260928 21:09:35.597834 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:09:35.598213 22065 external_mini_cluster.cc:1428] Setting key 86ca4df0f228f3a574e2d7e4715d491e
I20260928 21:09:35.598610 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:35.605646 5108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap starting.
Sep 28 21:09:35 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:35.608435 5108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:35.609359 5108 log.cc:824] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:35.611717 5108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: No bootstrap required, opened a new log
I20260928 21:09:35.613778 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:35.599995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.21.140.126:40549)
0928 21:09:35.600433 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:35.600451 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:35.600495 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:09:35.601291 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:35.601310 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:35.601684 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:35.602016 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:35.602987 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:35.603985 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:35.605184 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:35.605650 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:35.605823 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:35.608588 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:35.608595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:35.608611 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:35.608645 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:35.610605 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:35.611050 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:35.611056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:35.611063 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:35.611198 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:35.611458 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:35.611462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:35.611464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:35.611954 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:35.612162 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:35.612356 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:09:35.612645 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":151,"threads_started":1}
I20260928 21:09:35.615610 5108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:35.615854 5108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:35.615926 5108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1506ac661a2e463dac392a2c6c36478b, State: Initialized, Role: FOLLOWER
I20260928 21:09:35.616336 5108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:35.616457 5108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:35.616503 5108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:35.616588 5108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:35.617419 5108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:35.617726 5108 leader_election.cc:302] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1506ac661a2e463dac392a2c6c36478b; no voters:
I20260928 21:09:35.618031 5108 leader_election.cc:288] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:09:35.618144 5113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:35.618394 5113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 LEADER]: Becoming Leader. State: Replica: 1506ac661a2e463dac392a2c6c36478b, State: Running, Role: LEADER
I20260928 21:09:35.618745 5113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:35.619259 5108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:35.620537 5115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1506ac661a2e463dac392a2c6c36478b. Latest consensus state: current_term: 1 leader_uuid: "1506ac661a2e463dac392a2c6c36478b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } } }
I20260928 21:09:35.620561 5114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1506ac661a2e463dac392a2c6c36478b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } } }
I20260928 21:09:35.620690 5115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:35.620695 5114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:35.621057 5122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:35.624101 5122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:35.629848 5122 catalog_manager.cc:1347] Generated new cluster ID: 81f8bcf1ac9b4955ac8fc1d941a76312
I20260928 21:09:35.629913 5122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:35.649596 5122 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:09:35.650835 5122 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:35.664192 5122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Generated new TSK 0
I20260928 21:09:35.665020 5122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:09:35.702502 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40549
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:35.815689 5134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:35.816000 5134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:35.816082 5134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:35.820343 5134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:35.820432 5134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:35.820554 5134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:35.825814 5134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:35.826905 5134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:35.827945 5134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:09:35.837702 5134 server_base.cc:1034] running on GCE node
W20260928 21:09:35.837754 5139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:35.837767 5142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:35.837728 5140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:35.838491 5134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:35.839195 5134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:35.840389 5134 hybrid_clock.cc:648] HybridClock initialized: now 1790629775840359 us; error 42 us; skew 500 ppm
Sep 28 21:09:35 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629775, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:35.843277 5134 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:35.844136 5134 webserver.cc:460] Webserver started at http://127.21.140.65:46611/ using document root <none> and password file <none>
I20260928 21:09:35.844736 5134 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:35.844801 5134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:35.845005 5134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:35.847082 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "9272954df65543b9bb73add09a102a0e"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "4a43be82696dbf622e2e2f156ff99dd2"
server_key_iv: "2f0c4a7d61e6a364f97d9478cb00e45d"
server_key_version: "encryptionkey@0"
I20260928 21:09:35.847649 5134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "9272954df65543b9bb73add09a102a0e"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "4a43be82696dbf622e2e2f156ff99dd2"
server_key_iv: "2f0c4a7d61e6a364f97d9478cb00e45d"
server_key_version: "encryptionkey@0"
I20260928 21:09:35.851387 5134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:09:35.853744 5149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:35.854979 5134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:35.855104 5134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "9272954df65543b9bb73add09a102a0e"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "4a43be82696dbf622e2e2f156ff99dd2"
server_key_iv: "2f0c4a7d61e6a364f97d9478cb00e45d"
server_key_version: "encryptionkey@0"
I20260928 21:09:35.855211 5134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:35.884480 5134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:35.887661 5134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:35.887863 5134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:35.888463 5134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:35.889288 5134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:35.889338 5134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:35.889407 5134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:35.889441 5134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:35.898453 5134 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:35337
I20260928 21:09:35.898528 5263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:35337 every 8 connection(s)
I20260928 21:09:35.899443 5134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:35.899734 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5134
I20260928 21:09:35.899880 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:09:35.900218 22065 external_mini_cluster.cc:1428] Setting key 606994a8434795480404053f45d3b7f8
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Sep 28 21:09:35 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629775, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:35.919312 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:35.901524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:35037 (local address 127.21.140.126:40549)
0928 21:09:35.901676 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:35.901680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:35.902773 (+ 1093us) server_negotiation.cc:407] Connection header received
0928 21:09:35.904459 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:35.904465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:35.904528 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:35.904644 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:35.906470 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:35.907113 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:35.909041 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:35.909273 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:35.909354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:35.915158 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:35.915161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:35.915163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:35.915191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:35.916990 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:35.917601 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:35.917604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:35.917605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:35.917685 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:35.918117 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:35.918120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:35.918122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:35.918319 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:35.918433 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:35.919012 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 21:09:35.919145 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:09:35.920240 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:35.902051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:35037)
0928 21:09:35.902585 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:35.902648 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:35.904227 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:35.904884 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:35.904897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:35.905451 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:35.906262 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:35.906289 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:35.907269 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:35.907275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:35.908902 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:35.908916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:35.909445 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:35.909450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:35.909696 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:35.910400 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:35.910427 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:35.914918 (+ 4491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:35.917146 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:35.917151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:35.917165 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:35.917443 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:35.917800 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:35.917803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:35.917804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:35.917943 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:35.918433 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:35.918441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:35.918719 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 21:09:35.918954 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 21:09:35.919303 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260928 21:09:35.921701 5264 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40549
I20260928 21:09:35.922008 5264 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:35.922641 5264 heartbeater.cc:511] Master 127.21.140.126:40549 requested a full tablet report, sending...
I20260928 21:09:35.924463 5071 ts_manager.cc:194] Registered new tserver with Master: 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337)
I20260928 21:09:35.925807 5071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:35037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:09:35.931404 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40549
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:36.046535 5269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:36.046883 5269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:36.046931 5269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:36.050936 5269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:36.050992 5269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:36.051049 5269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:36.056213 5269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:36.057318 5269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:36.058331 5269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:36.068528 5274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:36.068612 5269 server_base.cc:1034] running on GCE node
W20260928 21:09:36.068617 5277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:36.068557 5275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:36.069173 5269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:36.069819 5269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:36.070997 5269 hybrid_clock.cc:648] HybridClock initialized: now 1790629776070965 us; error 45 us; skew 500 ppm
Sep 28 21:09:36 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:36.074391 5269 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:36.075284 5269 webserver.cc:460] Webserver started at http://127.21.140.66:32983/ using document root <none> and password file <none>
I20260928 21:09:36.075863 5269 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:36.075956 5269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:36.076226 5269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:36.078437 5269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "e5adf57374254ab3a2764b83c2823e9e"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "72b0d0c2dd6747942c43a2e85e2ca5a9"
server_key_iv: "4226cb81a27ade996fc24b7241c374d6"
server_key_version: "encryptionkey@0"
I20260928 21:09:36.078985 5269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "e5adf57374254ab3a2764b83c2823e9e"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "72b0d0c2dd6747942c43a2e85e2ca5a9"
server_key_iv: "4226cb81a27ade996fc24b7241c374d6"
server_key_version: "encryptionkey@0"
I20260928 21:09:36.082819 5269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:09:36.085299 5284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:36.086565 5269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:09:36.086712 5269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "e5adf57374254ab3a2764b83c2823e9e"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "72b0d0c2dd6747942c43a2e85e2ca5a9"
server_key_iv: "4226cb81a27ade996fc24b7241c374d6"
server_key_version: "encryptionkey@0"
I20260928 21:09:36.086835 5269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:36.105396 5269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:36.108496 5269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:36.108721 5269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:36.109301 5269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:36.110184 5269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:36.110255 5269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:36.110364 5269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:36.110417 5269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:36.119545 5269 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:46271
I20260928 21:09:36.119619 5398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:46271 every 8 connection(s)
I20260928 21:09:36.120688 5269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:09:36.128106 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5269
I20260928 21:09:36.128247 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:09:36.128573 22065 external_mini_cluster.cc:1428] Setting key 589afae8f74d6dbe066988c274068f83
Sep 28 21:09:36 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:36.135298 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.122678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:42803 (local address 127.21.140.126:40549)
0928 21:09:36.122840 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.122845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.123650 (+ 805us) server_negotiation.cc:407] Connection header received
0928 21:09:36.124945 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.124948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.124995 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.125090 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:36.126313 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.126847 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.128243 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.128558 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.128703 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.131513 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:36.131516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:36.131518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:36.131549 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:36.133058 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:36.133648 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:36.133652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:36.133655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:36.133738 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:36.134166 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:36.134171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:36.134177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:36.134354 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:36.134515 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.135010 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.135133 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:09:36.136083 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.123036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:42803)
0928 21:09:36.123497 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.123539 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.124751 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.125249 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.125255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.125591 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:36.126155 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.126171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.126966 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.126968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.128063 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.128075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.128753 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.128758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.128952 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.129554 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:36.129574 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:36.131333 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:36.133187 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:36.133193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:36.133208 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:36.133542 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:36.133844 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:36.133847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:36.133849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:36.134048 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:36.134516 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:36.134524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:36.134770 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.134952 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.135276 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260928 21:09:36.137245 5399 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40549
I20260928 21:09:36.137494 5399 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:09:36.138101 5399 heartbeater.cc:511] Master 127.21.140.126:40549 requested a full tablet report, sending...
I20260928 21:09:36.139209 5071 ts_manager.cc:194] Registered new tserver with Master: e5adf57374254ab3a2764b83c2823e9e (127.21.140.66:46271)
I20260928 21:09:36.139763 5071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:42803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:09:36.155149 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40549
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:36.266512 5404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:36.266788 5404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:36.266861 5404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:36.271004 5404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:36.271090 5404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:36.271173 5404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:36.276566 5404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:36.277681 5404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:36.278764 5404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:36.288619 5412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:36.288739 5409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:36.288619 5410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:36.288870 5404 server_base.cc:1034] running on GCE node
I20260928 21:09:36.289326 5404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:36.290019 5404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:36.291195 5404 hybrid_clock.cc:648] HybridClock initialized: now 1790629776291186 us; error 30 us; skew 500 ppm
Sep 28 21:09:36 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:36.293953 5404 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:36.294886 5404 webserver.cc:460] Webserver started at http://127.21.140.67:44215/ using document root <none> and password file <none>
I20260928 21:09:36.295464 5404 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:36.295560 5404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:36.295775 5404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:09:36.297843 5404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "f875d8bcc9dd4f90927b23ee2e048246"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "70dcfa2883c7584be7955559149e88a4"
server_key_iv: "29f63dee6ed3e7586a8120b2410a4397"
server_key_version: "encryptionkey@0"
I20260928 21:09:36.298517 5404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "f875d8bcc9dd4f90927b23ee2e048246"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "70dcfa2883c7584be7955559149e88a4"
server_key_iv: "29f63dee6ed3e7586a8120b2410a4397"
server_key_version: "encryptionkey@0"
I20260928 21:09:36.302713 5404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:09:36.305373 5419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:36.306800 5404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:09:36.306944 5404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "f875d8bcc9dd4f90927b23ee2e048246"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "70dcfa2883c7584be7955559149e88a4"
server_key_iv: "29f63dee6ed3e7586a8120b2410a4397"
server_key_version: "encryptionkey@0"
I20260928 21:09:36.307070 5404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:36.328245 5404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:36.331368 5404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:36.331568 5404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:36.332162 5404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:36.333003 5404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:36.333071 5404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:36.333148 5404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:36.333190 5404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:36.342309 5404 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35327
I20260928 21:09:36.342325 5533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35327 every 8 connection(s)
I20260928 21:09:36.343178 5404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:09:36.351590 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5404
I20260928 21:09:36.351738 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:09:36.352027 22065 external_mini_cluster.cc:1428] Setting key 5af6d002a9ed7261cdbf7f733eb4a28e
Sep 28 21:09:36 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:36.357064 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.344924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:42301 (local address 127.21.140.126:40549)
0928 21:09:36.345076 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.345080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.345927 (+ 847us) server_negotiation.cc:407] Connection header received
0928 21:09:36.347305 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.347308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.347369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.347485 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:36.348647 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.349178 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.350420 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.350643 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.350764 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.353597 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:36.353600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:36.353602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:36.353627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:36.355200 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:36.355653 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:36.355655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:36.355657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:36.355729 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:36.356100 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:36.356102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:36.356103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:36.356292 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:36.356368 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.356853 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.356946 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:09:36.358057 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.345312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:42301)
0928 21:09:36.345776 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.345817 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.347085 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.347658 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.347665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.348033 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:36.348495 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.348507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.349332 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.349334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.350287 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.350296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.350803 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.350807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.350976 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.351589 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:36.351604 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:36.353440 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:36.355302 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:36.355307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:36.355319 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:36.355566 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:36.355839 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:36.355842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:36.355843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:36.356000 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:36.356391 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:36.356396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:36.356639 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.356807 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.357081 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260928 21:09:36.359344 5534 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40549
I20260928 21:09:36.359591 5534 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:36.360124 5534 heartbeater.cc:511] Master 127.21.140.126:40549 requested a full tablet report, sending...
I20260928 21:09:36.361232 5071 ts_manager.cc:194] Registered new tserver with Master: f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327)
I20260928 21:09:36.361819 5071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:42301
I20260928 21:09:36.366024 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:09:36.380147 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.368167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.21.140.126:40549)
0928 21:09:36.368281 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.368291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.368566 (+ 275us) server_negotiation.cc:407] Connection header received
0928 21:09:36.368610 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.368612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.368653 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.368724 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:36.369419 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.370060 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.370899 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.371118 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.371199 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.377037 (+ 5838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:36.377040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:36.377057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:36.377083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:36.378535 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:36.378971 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:36.378974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:36.378975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:36.379059 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:36.379354 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:36.379357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:36.379359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:36.379615 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:36.379725 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.379930 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.380031 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:09:36.383165 5071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:09:36.386368 5071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:09:36.402091 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.395873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.21.140.66:46271)
0928 21:09:36.396917 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.396930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.396951 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:09:36.397024 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.397029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.397209 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.397626 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:36.397797 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.398768 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.400173 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.401092 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.401216 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.401369 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.401763 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.401865 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":97,"threads_started":1}
I20260928 21:09:36.402087 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.395211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.0.0.1:46084)
0928 21:09:36.395630 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.395682 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.395805 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.396605 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.396610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.396632 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:36.396784 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.396793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.398494 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.398496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.400162 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.400174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.401260 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.401264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.401505 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.401586 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.401713 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.401891 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":273,"thread_start_us":117,"threads_started":1}
I20260928 21:09:36.402140 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.395446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46271 (local address 127.0.0.1:37414)
0928 21:09:36.396174 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.396190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.396287 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.397368 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.397371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.397389 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:36.397485 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.397490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.398908 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.398911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.400038 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.400048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.401260 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.401264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.401492 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.401585 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.401713 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.401891 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":141,"thread_start_us":91,"threads_started":1}
I20260928 21:09:36.402797 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.395976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35327 (local address 127.0.0.1:51778)
0928 21:09:36.396453 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.396464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.396575 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.397253 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.397256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.397274 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:36.397367 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.397374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.398432 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.398438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.400162 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.400174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.402413 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.402417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.402505 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.402529 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.402576 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.402650 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":162,"thread_start_us":117,"threads_started":1}
I20260928 21:09:36.403290 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.395388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.21.140.65:35337)
0928 21:09:36.396063 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.396078 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.396103 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:09:36.396201 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.396207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.396430 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.396950 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:36.397105 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.398326 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.400309 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.401092 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.401216 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.401369 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.402992 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.403123 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":81,"threads_started":1}
I20260928 21:09:36.403688 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.396368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.21.140.67:35327)
0928 21:09:36.396837 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.396850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.396872 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:09:36.396946 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.396951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.397119 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.397275 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:36.397462 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.398289 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.401446 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.402247 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.403216 (+ 969us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.403394 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.403449 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.403551 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":147,"threads_started":1}
I20260928 21:09:36.405874 5198 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d (DEFAULT_TABLE table=test-table [id=c3b5d63a3f534886b61460182d7b62ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:36.406039 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d (DEFAULT_TABLE table=test-table [id=c3b5d63a3f534886b61460182d7b62ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:36.406607 5468 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d (DEFAULT_TABLE table=test-table [id=c3b5d63a3f534886b61460182d7b62ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:36.407120 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:36.407107 5198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:36.407528 5468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:36.413627 5551 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Bootstrap starting.
I20260928 21:09:36.415102 5552 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Bootstrap starting.
I20260928 21:09:36.415983 5551 tablet_bootstrap.cc:654] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:36.416983 5552 tablet_bootstrap.cc:654] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:36.417189 5551 log.cc:824] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:36.418061 5553 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Bootstrap starting.
I20260928 21:09:36.418339 5552 log.cc:824] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:36.420499 5552 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: No bootstrap required, opened a new log
I20260928 21:09:36.420518 5553 tablet_bootstrap.cc:654] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:36.420735 5552 ts_tablet_manager.cc:1397] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:09:36.421420 5553 log.cc:824] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:36.423141 5551 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: No bootstrap required, opened a new log
I20260928 21:09:36.423347 5551 ts_tablet_manager.cc:1397] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 21:09:36.423403 5553 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: No bootstrap required, opened a new log
I20260928 21:09:36.423563 5553 ts_tablet_manager.cc:1397] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:09:36.424304 5552 raft_consensus.cc:348] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.424582 5552 raft_consensus.cc:374] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:36.424635 5552 raft_consensus.cc:729] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f875d8bcc9dd4f90927b23ee2e048246, State: Initialized, Role: FOLLOWER
I20260928 21:09:36.426096 5552 consensus_queue.cc:260] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.426813 5534 heartbeater.cc:503] Master 127.21.140.126:40549 was elected leader, sending a full tablet report...
I20260928 21:09:36.427177 5552 ts_tablet_manager.cc:1428] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:09:36.427294 5551 raft_consensus.cc:348] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.427294 5553 raft_consensus.cc:348] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.427546 5553 raft_consensus.cc:374] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:36.427546 5551 raft_consensus.cc:374] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:36.427603 5553 raft_consensus.cc:729] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5adf57374254ab3a2764b83c2823e9e, State: Initialized, Role: FOLLOWER
I20260928 21:09:36.427596 5551 raft_consensus.cc:729] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9272954df65543b9bb73add09a102a0e, State: Initialized, Role: FOLLOWER
I20260928 21:09:36.428084 5551 consensus_queue.cc:260] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.428084 5553 consensus_queue.cc:260] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.428719 5264 heartbeater.cc:503] Master 127.21.140.126:40549 was elected leader, sending a full tablet report...
I20260928 21:09:36.429014 5553 ts_tablet_manager.cc:1428] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:09:36.429356 5399 heartbeater.cc:503] Master 127.21.140.126:40549 was elected leader, sending a full tablet report...
I20260928 21:09:36.430008 5551 ts_tablet_manager.cc:1428] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:09:36.549669 5559 raft_consensus.cc:482] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:36.549916 5559 raft_consensus.cc:504] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.551119 5559 leader_election.cc:288] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337), f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327)
I20260928 21:09:36.556419 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.551528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:39709 (local address 127.21.140.65:35337)
0928 21:09:36.551682 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.551686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.551708 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:09:36.552083 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.552092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.552137 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.552208 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:36.552558 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.553384 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.555088 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.555686 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.555821 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.555872 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.556220 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.556294 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:09:36.556419 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.551925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:37787 (local address 127.21.140.67:35327)
0928 21:09:36.552127 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.552132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.552330 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:09:36.552514 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.552521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.552581 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.552672 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:36.552901 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.553629 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.555087 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.555716 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.555822 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.555872 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.556221 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.556293 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 21:09:36.556433 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.551425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.21.140.66:39709)
0928 21:09:36.551640 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.551657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.551759 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.552241 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.552246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.552273 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:36.552424 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.552431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.553518 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.553521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.554938 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.554950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.555861 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.555865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.556031 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.556083 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.556173 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.556257 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":63}
I20260928 21:09:36.556443 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.551749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35327 (local address 127.21.140.66:37787)
0928 21:09:36.552217 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.552230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:36.552337 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:36.552689 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:36.552691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:36.552707 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:36.552796 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.552799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.553731 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.553734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.554962 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:36.554969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.555861 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.555865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:36.556026 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.556083 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:09:36.556196 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 21:09:36.556257 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":115,"spinlock_wait_cycles":2304,"thread_start_us":162,"threads_started":1}
I20260928 21:09:36.557226 5218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0bf4f004ae41eba6cf63ff4ddf8a8d" candidate_uuid: "e5adf57374254ab3a2764b83c2823e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9272954df65543b9bb73add09a102a0e" is_pre_election: true
I20260928 21:09:36.557227 5488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0bf4f004ae41eba6cf63ff4ddf8a8d" candidate_uuid: "e5adf57374254ab3a2764b83c2823e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f875d8bcc9dd4f90927b23ee2e048246" is_pre_election: true
I20260928 21:09:36.557509 5488 raft_consensus.cc:2454] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5adf57374254ab3a2764b83c2823e9e in term 0.
I20260928 21:09:36.557521 5218 raft_consensus.cc:2454] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5adf57374254ab3a2764b83c2823e9e in term 0.
I20260928 21:09:36.558002 5287 leader_election.cc:302] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5adf57374254ab3a2764b83c2823e9e, f875d8bcc9dd4f90927b23ee2e048246; no voters:
I20260928 21:09:36.558336 5559 raft_consensus.cc:2781] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:09:36.558435 5559 raft_consensus.cc:482] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:09:36.558494 5559 raft_consensus.cc:3037] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:36.559594 5559 raft_consensus.cc:504] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.559927 5559 leader_election.cc:288] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [CANDIDATE]: Term 1 election: Requested vote from peers 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337), f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327)
I20260928 21:09:36.560495 5488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0bf4f004ae41eba6cf63ff4ddf8a8d" candidate_uuid: "e5adf57374254ab3a2764b83c2823e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f875d8bcc9dd4f90927b23ee2e048246"
I20260928 21:09:36.560722 5488 raft_consensus.cc:3037] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:36.561919 5488 raft_consensus.cc:2454] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5adf57374254ab3a2764b83c2823e9e in term 1.
I20260928 21:09:36.562216 5218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0bf4f004ae41eba6cf63ff4ddf8a8d" candidate_uuid: "e5adf57374254ab3a2764b83c2823e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9272954df65543b9bb73add09a102a0e"
I20260928 21:09:36.562269 5287 leader_election.cc:302] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5adf57374254ab3a2764b83c2823e9e, f875d8bcc9dd4f90927b23ee2e048246; no voters:
I20260928 21:09:36.562345 5218 raft_consensus.cc:3037] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:36.562446 5559 raft_consensus.cc:2781] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:36.562693 5559 raft_consensus.cc:686] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 1 LEADER]: Becoming Leader. State: Replica: e5adf57374254ab3a2764b83c2823e9e, State: Running, Role: LEADER
I20260928 21:09:36.563030 5559 consensus_queue.cc:237] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:36.563730 5218 raft_consensus.cc:2454] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5adf57374254ab3a2764b83c2823e9e in term 1.
I20260928 21:09:36.565846 5071 catalog_manager.cc:5495] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to e5adf57374254ab3a2764b83c2823e9e (127.21.140.66). New cstate: current_term: 1 leader_uuid: "e5adf57374254ab3a2764b83c2823e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } health_report { overall_health: HEALTHY } } }
I20260928 21:09:36.591552 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.587922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.21.140.66:46271)
0928 21:09:36.588060 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:36.588064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:36.588080 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:09:36.588210 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:36.588212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:36.588267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:36.588384 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:36.588681 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.589190 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.590088 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:36.590311 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:36.590417 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:36.590743 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:36.591152 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:09:36.591300 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:36.591341 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:09:36.591395 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 21:09:36.594149 5535 tablet.cc:2369] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:09:36.597460 5488 raft_consensus.cc:1261] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Refusing update from remote peer e5adf57374254ab3a2764b83c2823e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:36.597540 5218 raft_consensus.cc:1261] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Refusing update from remote peer e5adf57374254ab3a2764b83c2823e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:36.598172 5561 consensus_queue.cc:1035] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:36.598358 5559 consensus_queue.cc:1035] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:36.605160 5564 mvcc.cc:205] Tried to move back new op lower bound from 7334419564935340032 to 7334419564804681728. Current Snapshot: MvccSnapshot[applied={T|T < 7334419564935340032}]
I20260928 21:09:36.608264 5566 mvcc.cc:205] Tried to move back new op lower bound from 7334419564935340032 to 7334419564804681728. Current Snapshot: MvccSnapshot[applied={T|T < 7334419564935340032}]
I20260928 21:09:36.618042 5071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:36.618232 5071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:36.620715 5071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Sending DeleteTablet for 3 replicas of tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d
I20260928 21:09:36.621502 5333 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:37414
I20260928 21:09:36.621589 5198 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:46084
I20260928 21:09:36.621989 5574 tablet_replica.cc:330] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: stopping tablet replica
I20260928 21:09:36.622026 5468 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:51778
I20260928 21:09:36.622516 5576 tablet_replica.cc:330] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: stopping tablet replica
I20260928 21:09:36.622690 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5134
I20260928 21:09:36.622671 5574 raft_consensus.cc:2229] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:09:36.622932 5576 raft_consensus.cc:2229] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:36.623317 5576 raft_consensus.cc:2258] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:36.625758 5576 ts_tablet_manager.cc:1905] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:36.626269 5574 raft_consensus.cc:2258] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:36.627731 5574 ts_tablet_manager.cc:1905] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:36.628701 5576 ts_tablet_manager.cc:1918] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:36.628804 5576 log.cc:1196] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/9c0bf4f004ae41eba6cf63ff4ddf8a8d
I20260928 21:09:36.629165 5576 ts_tablet_manager.cc:1939] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P f875d8bcc9dd4f90927b23ee2e048246: Deleting consensus metadata
I20260928 21:09:36.630447 5057 catalog_manager.cc:4841] TS f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327): tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d (table test-table [id=c3b5d63a3f534886b61460182d7b62ae]) successfully deleted
I20260928 21:09:36.630913 5574 ts_tablet_manager.cc:1918] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:36.631003 5574 log.cc:1196] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/9c0bf4f004ae41eba6cf63ff4ddf8a8d
W20260928 21:09:36.631112 5057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:35337 (error 108)
I20260928 21:09:36.631340 5574 ts_tablet_manager.cc:1939] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P e5adf57374254ab3a2764b83c2823e9e: Deleting consensus metadata
I20260928 21:09:36.632167 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5269
I20260928 21:09:36.632189 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.631882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.0.0.1:46088)
0928 21:09:36.632023 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.632107 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 21:09:36.633674 5057 catalog_manager.cc:4584] TS 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d: Network error: Client connection negotiation failed: client connection to 127.21.140.65:35337: connect: Connection refused (error 111)
I20260928 21:09:36.638957 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5404
I20260928 21:09:36.639330 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:36.638985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46271 (local address 127.0.0.1:37430)
0928 21:09:36.639178 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:36.639239 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 21:09:36.644598 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5039
I20260928 21:09:56.651489 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40549
--webserver_interface=127.21.140.126
--webserver_port=41441
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:09:56.792325 5578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:56.792737 5578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:56.792824 5578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:56.797354 5578 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:09:56.797459 5578 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:09:56.797478 5578 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:09:56.797493 5578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:09:56.797514 5578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:56.797533 5578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:56.797550 5578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:09:56.797567 5578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:09:56.804124 5578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:56.805794 5578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:56.807338 5578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:56.826170 5585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:56.826170 5584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:56.826515 5578 server_base.cc:1034] running on GCE node
W20260928 21:09:56.826172 5587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:56.827533 5578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:56.829288 5578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:56.830610 5578 hybrid_clock.cc:648] HybridClock initialized: now 1790629796830570 us; error 76 us; skew 500 ppm
Sep 28 21:09:56 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629796, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:56.839058 5578 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:09:56.840582 5578 webserver.cc:460] Webserver started at http://127.21.140.126:41441/ using document root <none> and password file <none>
I20260928 21:09:56.841434 5578 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:56.841564 5578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:56.847090 5578 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:09:56.850337 5594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:56.852358 5578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:09:56.852595 5578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "1506ac661a2e463dac392a2c6c36478b"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "ace067dad802d98f5ec8fdce5b776334"
server_key_iv: "f2454b64c1a1768b1937a209f42a776d"
server_key_version: "encryptionkey@0"
I20260928 21:09:56.853468 5578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:56.875093 5578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:56.880141 5578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:56.880472 5578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:56.890623 5578 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40549
I20260928 21:09:56.891011 5647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40549 every 8 connection(s)
I20260928 21:09:56.892343 5578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:09:56.893445 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5578
I20260928 21:09:56.894111 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:35337
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:40549
--webserver_port=46611
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:56.900071 5648 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:09:56.902551 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap starting.
I20260928 21:09:56.924638 5648 log.cc:824] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:56.936100 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:56.936666 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap complete.
I20260928 21:09:56.941869 5648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:56.942550 5648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1506ac661a2e463dac392a2c6c36478b, State: Initialized, Role: FOLLOWER
I20260928 21:09:56.943336 5648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:56.943511 5648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:09:56.943624 5648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:09:56.943778 5648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:09:56.946771 5648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:56.947361 5648 leader_election.cc:302] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1506ac661a2e463dac392a2c6c36478b; no voters:
I20260928 21:09:56.947904 5648 leader_election.cc:288] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:09:56.948029 5652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:09:56.948246 5652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 LEADER]: Becoming Leader. State: Replica: 1506ac661a2e463dac392a2c6c36478b, State: Running, Role: LEADER
I20260928 21:09:56.948659 5652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:09:56.949712 5648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:09:56.950678 5654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1506ac661a2e463dac392a2c6c36478b. Latest consensus state: current_term: 2 leader_uuid: "1506ac661a2e463dac392a2c6c36478b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } } }
I20260928 21:09:56.950681 5653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1506ac661a2e463dac392a2c6c36478b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } } }
I20260928 21:09:56.950879 5654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:56.950879 5653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: This master's current role is: LEADER
I20260928 21:09:56.951375 5656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:09:56.957911 5656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:09:56.958850 5656 catalog_manager.cc:1259] Loaded cluster ID: 81f8bcf1ac9b4955ac8fc1d941a76312
I20260928 21:09:56.958942 5656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:09:56.962164 5656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:09:56.963183 5656 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Loaded TSK: 0
I20260928 21:09:56.982941 5656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Generated new TSK 1
I20260928 21:09:56.983371 5656 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:09:57.044822 5650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:57.045279 5650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:57.045373 5650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:57.050297 5650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:57.050431 5650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:57.050541 5650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:09:57.057427 5650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:35337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=46611
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:57.059208 5650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:57.060850 5650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:57.075217 5675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:57.075397 5676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:57.075217 5678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:57.076468 5650 server_base.cc:1034] running on GCE node
I20260928 21:09:57.077206 5650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:57.078272 5650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:57.079576 5650 hybrid_clock.cc:648] HybridClock initialized: now 1790629797079536 us; error 72 us; skew 500 ppm
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:57.083675 5650 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:09:57.085258 5650 webserver.cc:460] Webserver started at http://127.21.140.65:46611/ using document root <none> and password file <none>
I20260928 21:09:57.086584 5650 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:57.086768 5650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:57.092739 5650 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 21:09:57.096261 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.098451 5650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:09:57.098660 5650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "9272954df65543b9bb73add09a102a0e"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "4a43be82696dbf622e2e2f156ff99dd2"
server_key_iv: "2f0c4a7d61e6a364f97d9478cb00e45d"
server_key_version: "encryptionkey@0"
I20260928 21:09:57.099526 5650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:57.116701 5650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:57.121894 5650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:57.122257 5650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:57.123214 5650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:57.124845 5692 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:09:57.130954 5650 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:09:57.131086 5650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260928 21:09:57.131191 5650 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:09:57.133760 5650 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:09:57.133908 5650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260928 21:09:57.134318 5692 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Bootstrap starting.
I20260928 21:09:57.148257 5650 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:35337
I20260928 21:09:57.148380 5800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:35337 every 8 connection(s)
I20260928 21:09:57.149809 5650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:09:57.154589 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5650
I20260928 21:09:57.155392 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:46271
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:40549
--webserver_port=32983
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:09:57.160718 5692 log.cc:824] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:57.174813 5692 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:09:57.175558 5692 tablet_bootstrap.cc:492] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Bootstrap complete.
I20260928 21:09:57.177337 5692 ts_tablet_manager.cc:1397] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Time spent bootstrapping tablet: real 0.043s user 0.028s sys 0.011s
I20260928 21:09:57.180119 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.153489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:43185 (local address 127.21.140.126:40549)
0928 21:09:57.154171 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.154201 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.155926 (+ 1725us) server_negotiation.cc:407] Connection header received
0928 21:09:57.157268 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.157289 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.157951 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.158550 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:57.160602 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.162662 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.165611 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.166853 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.167175 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.171792 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:57.171814 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:57.171839 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:57.171895 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:57.174752 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.175831 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.175841 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.175848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.176087 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.176649 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.176653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.176655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.177322 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:57.177593 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.178481 (+ 888us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.178886 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":239,"threads_started":1}
I20260928 21:09:57.180120 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.153486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:43185)
0928 21:09:57.155159 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.155229 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.156562 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.158703 (+ 2141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.158716 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.159390 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:57.160256 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.160297 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.163055 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.163061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.165110 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.165135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.167162 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.167168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.167466 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.168543 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:57.168577 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:57.171357 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:57.175100 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.175112 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.175136 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.175641 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.176242 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.176249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.176255 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.176520 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.177509 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:57.177518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:57.177991 (+ 473us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.178276 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.178791 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":217,"threads_started":1}
I20260928 21:09:57.181514 5692 raft_consensus.cc:348] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:57.182317 5692 raft_consensus.cc:729] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9272954df65543b9bb73add09a102a0e, State: Initialized, Role: FOLLOWER
I20260928 21:09:57.183018 5692 consensus_queue.cc:260] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } }
I20260928 21:09:57.184376 5801 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40549
I20260928 21:09:57.184655 5692 ts_tablet_manager.cc:1428] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260928 21:09:57.184725 5801 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:57.185516 5801 heartbeater.cc:511] Master 127.21.140.126:40549 requested a full tablet report, sending...
I20260928 21:09:57.189759 5611 ts_manager.cc:194] Registered new tserver with Master: 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337)
I20260928 21:09:57.195222 5611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:43185
I20260928 21:09:57.199074 5801 heartbeater.cc:503] Master 127.21.140.126:40549 was elected leader, sending a full tablet report...
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629796, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:09:57.208847 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.194686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.0.0.1:53974)
0928 21:09:57.195740 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.195802 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.196068 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.196753 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.196760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.197254 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:57.197411 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.197419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.198581 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.198585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.199401 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.199421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.200215 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.200218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.200394 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.201601 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:57.201619 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:57.203777 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:57.206381 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.206400 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.206408 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.206756 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.207164 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.207168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.207170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.207284 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.207939 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:57.207947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:57.208119 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.208292 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.208619 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":217,"threads_started":1}
I20260928 21:09:57.208858 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.194770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.21.140.65:35337)
0928 21:09:57.195414 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.195430 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.195981 (+ 551us) server_negotiation.cc:407] Connection header received
0928 21:09:57.196291 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.196299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.196574 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.196752 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:57.197566 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.198441 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.199549 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.200076 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.200213 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.204023 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:57.204038 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:57.204051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:57.204084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:57.206207 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.206924 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.206931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.206933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.207037 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.207399 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.207403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.207405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.207772 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:57.207952 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.208394 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.208619 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":499,"thread_start_us":180,"threads_started":1}
I20260928 21:09:57.210464 5735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:53974
I20260928 21:09:57.211695 5810 tablet_replica.cc:330] stopping tablet replica
I20260928 21:09:57.212182 5810 raft_consensus.cc:2229] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:57.212558 5810 raft_consensus.cc:2258] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:57.214192 5810 ts_tablet_manager.cc:1905] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:57.220739 5810 ts_tablet_manager.cc:1918] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:09:57.220929 5810 log.cc:1196] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal/wals/9c0bf4f004ae41eba6cf63ff4ddf8a8d
I20260928 21:09:57.221240 5810 ts_tablet_manager.cc:1939] T 9c0bf4f004ae41eba6cf63ff4ddf8a8d P 9272954df65543b9bb73add09a102a0e: Deleting consensus metadata
I20260928 21:09:57.224123 5597 catalog_manager.cc:4841] TS 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337): tablet 9c0bf4f004ae41eba6cf63ff4ddf8a8d (table test-table [id=c3b5d63a3f534886b61460182d7b62ae]) successfully deleted
W20260928 21:09:57.318553 5805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:57.318964 5805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:57.319016 5805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:57.323602 5805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:57.323695 5805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:57.323768 5805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:09:57.330186 5805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:46271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=32983
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:57.331725 5805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:57.333078 5805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:57.345345 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:57.345371 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:57.345345 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:57.345777 5805 server_base.cc:1034] running on GCE node
I20260928 21:09:57.346292 5805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:57.347059 5805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:57.348291 5805 hybrid_clock.cc:648] HybridClock initialized: now 1790629797348255 us; error 55 us; skew 500 ppm
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:57.351666 5805 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:09:57.352730 5805 webserver.cc:460] Webserver started at http://127.21.140.66:32983/ using document root <none> and password file <none>
I20260928 21:09:57.353426 5805 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:57.353543 5805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:57.358029 5805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:09:57.360342 5825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.361928 5805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:57.362185 5805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "e5adf57374254ab3a2764b83c2823e9e"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "72b0d0c2dd6747942c43a2e85e2ca5a9"
server_key_iv: "4226cb81a27ade996fc24b7241c374d6"
server_key_version: "encryptionkey@0"
I20260928 21:09:57.362849 5805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:57.385871 5805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:57.389411 5805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:57.389653 5805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:57.390424 5805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:57.391361 5805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:57.391438 5805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.391505 5805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:57.391556 5805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.400914 5805 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:46271
I20260928 21:09:57.400933 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:46271 every 8 connection(s)
I20260928 21:09:57.402155 5805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:09:57.406582 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5805
I20260928 21:09:57.407119 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:35327
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:40549
--webserver_port=44215
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:57.421830 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.404246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:51513 (local address 127.21.140.126:40549)
0928 21:09:57.404464 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.404468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.405632 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 21:09:57.407427 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.407431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.407494 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.407549 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:57.409245 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.410332 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.412650 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.412929 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.413043 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.416918 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:57.416921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:57.416923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:57.416952 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:57.418672 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.419556 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.419560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.419562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.419669 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.420209 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.420212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.420214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.420443 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:57.420593 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.421418 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.421667 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 21:09:57.423050 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.404698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:51513)
0928 21:09:57.405303 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.405474 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.407100 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.408024 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.408032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.408456 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:57.409045 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.409061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.410526 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.410534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.412431 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.412451 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.413157 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.413165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.413461 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.414299 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:57.414330 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:57.416662 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:57.418838 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.418850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.418873 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.419371 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.419797 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.419803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.419809 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.420066 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.420604 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:57.420614 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:57.421063 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.421353 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.421790 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":135,"threads_started":1}
I20260928 21:09:57.424475 5940 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40549
I20260928 21:09:57.424767 5940 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:57.425312 5940 heartbeater.cc:511] Master 127.21.140.126:40549 requested a full tablet report, sending...
I20260928 21:09:57.426604 5611 ts_manager.cc:194] Registered new tserver with Master: e5adf57374254ab3a2764b83c2823e9e (127.21.140.66:46271)
I20260928 21:09:57.427486 5611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:51513
W20260928 21:09:57.535986 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:09:57.536259 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:09:57.536303 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:09:57.540400 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:09:57.540462 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:09:57.540539 5943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:09:57.545953 5943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:35327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=44215
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:09:57.547040 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:09:57.548040 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:09:57.557852 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:57.557905 5943 server_base.cc:1034] running on GCE node
W20260928 21:09:57.557865 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:09:57.557925 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:09:57.558509 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:09:57.559173 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:09:57.560369 5943 hybrid_clock.cc:648] HybridClock initialized: now 1790629797560335 us; error 52 us; skew 500 ppm
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:09:57.563285 5943 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:09:57.564271 5943 webserver.cc:460] Webserver started at http://127.21.140.67:44215/ using document root <none> and password file <none>
I20260928 21:09:57.564850 5943 fs_manager.cc:359] Metadata directory not provided
I20260928 21:09:57.564958 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:09:57.569224 5943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:57.571523 5958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.572820 5943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:09:57.572954 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "f875d8bcc9dd4f90927b23ee2e048246"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "70dcfa2883c7584be7955559149e88a4"
server_key_iv: "29f63dee6ed3e7586a8120b2410a4397"
server_key_version: "encryptionkey@0"
I20260928 21:09:57.573504 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:09:57.601325 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:09:57.604844 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:09:57.605059 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:09:57.605719 5943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:09:57.606617 5943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:09:57.606695 5943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.606763 5943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:09:57.606814 5943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:09:57.616355 5943 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35327
I20260928 21:09:57.616384 6072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35327 every 8 connection(s)
I20260928 21:09:57.617331 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:09:57.625830 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 5943
Sep 28 21:09:57 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:09:57.631657 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.619145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:40387 (local address 127.21.140.126:40549)
0928 21:09:57.619317 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.619321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.620096 (+ 775us) server_negotiation.cc:407] Connection header received
0928 21:09:57.621495 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.621498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.621554 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.621636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:57.622813 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.623353 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.624545 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.624783 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.624899 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.627855 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:57.627858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:57.627859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:57.627878 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:57.629330 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.629854 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.629859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.629864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.629952 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.630384 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.630390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.630394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.630591 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:57.630669 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.631335 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.631468 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 21:09:57.632467 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.619500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:40387)
0928 21:09:57.619936 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.619978 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.621280 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.621804 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.621811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.622191 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:09:57.622663 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.622677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.623492 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.623498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.624410 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.624419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.624948 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.624952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.625117 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.625723 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:09:57.625739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:09:57.627693 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:09:57.629442 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.629448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.629464 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.629762 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.630090 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:09:57.630093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:09:57.630095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:09:57.630289 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:09:57.630702 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:09:57.630708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:09:57.631063 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.631295 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.631599 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260928 21:09:57.633770 6073 heartbeater.cc:347] Connected to a master server at 127.21.140.126:40549
I20260928 21:09:57.634039 6073 heartbeater.cc:464] Registering TS with master...
I20260928 21:09:57.634536 6073 heartbeater.cc:511] Master 127.21.140.126:40549 requested a full tablet report, sending...
I20260928 21:09:57.635195 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.626584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.21.140.126:40549)
0928 21:09:57.626887 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.626890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.626999 (+ 109us) server_negotiation.cc:407] Connection header received
0928 21:09:57.627170 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.627174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.627222 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.627305 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:57.628184 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.628695 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.629565 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.629792 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.629921 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.631472 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:57.631476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:57.631478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:57.631499 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:57.633434 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.633953 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.633956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.633958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.634071 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.634350 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.634354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.634356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.634530 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:57.634746 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.634914 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.635038 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":73,"threads_started":1}
I20260928 21:09:57.635617 5611 ts_manager.cc:194] Registered new tserver with Master: f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327)
I20260928 21:09:57.636377 5611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:40387
I20260928 21:09:57.636919 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:09:57.640945 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.637496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.21.140.126:40549)
0928 21:09:57.637640 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.637644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.637820 (+ 176us) server_negotiation.cc:407] Connection header received
0928 21:09:57.638015 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.638020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.638130 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.638235 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:57.638461 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.638952 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.639798 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.640036 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.640150 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.640443 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:57.640570 (+ 127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 21:09:57.640830 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54028: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:09:57.641026 6077 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54028: token expired
W20260928 21:09:57.641323 5537 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.140.126:40549)
I20260928 21:09:57.648037 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.641685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.21.140.126:40549)
0928 21:09:57.641806 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.641811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.641826 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:09:57.641927 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.641936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.642005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.642084 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:09:57.642659 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.643102 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.643850 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.644045 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.644126 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.645291 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:09:57.645294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:09:57.645295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:09:57.645316 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:09:57.646751 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.647117 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.647120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.647122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.647196 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:09:57.647411 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:09:57.647413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:09:57.647414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:09:57.647557 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:09:57.647689 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.647858 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.647939 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:09:57.650420 5610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:09:57.654136 5610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:09:57.662889 5735 tablet_service.cc:1467] Processing CreateTablet for tablet a13fd378576947a786eb1546e9c7b3f8 (DEFAULT_TABLE table=test-table [id=7abe5950d2ba44d3ae807d35ceb0f360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:57.663620 5735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fd378576947a786eb1546e9c7b3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:57.666181 5692 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Bootstrap starting.
I20260928 21:09:57.667104 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.661858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35327 (local address 127.0.0.1:40264)
0928 21:09:57.662164 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.662410 (+ 246us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.662506 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.663445 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.663448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.663462 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:57.663608 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.663617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.664635 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.664640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.665449 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.665458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.666598 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.666602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.666692 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.666739 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.666846 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.666926 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":140,"threads_started":1}
I20260928 21:09:57.667102 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.662412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.21.140.67:35327)
0928 21:09:57.662717 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.662729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.662758 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:09:57.662822 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.662825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.663079 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.663240 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:57.663741 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.664511 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.665585 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.666485 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.666598 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.666819 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.666874 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.666960 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260928 21:09:57.667205 5692 tablet_bootstrap.cc:654] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:57.669005 5692 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: No bootstrap required, opened a new log
I20260928 21:09:57.669108 5692 ts_tablet_manager.cc:1397] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:09:57.669543 5692 raft_consensus.cc:348] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.669667 5692 raft_consensus.cc:374] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:57.669705 5692 raft_consensus.cc:729] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9272954df65543b9bb73add09a102a0e, State: Initialized, Role: FOLLOWER
I20260928 21:09:57.669826 5692 consensus_queue.cc:260] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.670231 5692 ts_tablet_manager.cc:1428] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:09:57.670513 6007 tablet_service.cc:1467] Processing CreateTablet for tablet a13fd378576947a786eb1546e9c7b3f8 (DEFAULT_TABLE table=test-table [id=7abe5950d2ba44d3ae807d35ceb0f360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:57.671562 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.661711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46271 (local address 127.0.0.1:46218)
0928 21:09:57.661856 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.661872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.661943 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.667006 (+ 5063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.667011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.667029 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:57.667146 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.667155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.668809 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.668812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.669891 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.669904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.671138 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.671144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.671262 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.671287 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.671339 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.671417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 21:09:57.671749 6007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fd378576947a786eb1546e9c7b3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:57.672209 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.662159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.21.140.66:46271)
0928 21:09:57.666441 (+ 4282us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.666456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.666526 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:09:57.666617 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.666625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.666840 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.667315 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:57.667519 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.668672 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.670073 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.670982 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.671732 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.671891 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.671944 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.672031 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4175,"thread_start_us":106,"threads_started":1}
I20260928 21:09:57.675120 5874 tablet_service.cc:1467] Processing CreateTablet for tablet a13fd378576947a786eb1546e9c7b3f8 (DEFAULT_TABLE table=test-table [id=7abe5950d2ba44d3ae807d35ceb0f360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:09:57.676179 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fd378576947a786eb1546e9c7b3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:09:57.680024 6086 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Bootstrap starting.
I20260928 21:09:57.682426 6086 tablet_bootstrap.cc:654] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:57.683543 6086 log.cc:824] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:57.683561 6087 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Bootstrap starting.
I20260928 21:09:57.685400 6086 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: No bootstrap required, opened a new log
I20260928 21:09:57.685614 6086 ts_tablet_manager.cc:1397] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:09:57.685878 6087 tablet_bootstrap.cc:654] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Neither blocks nor log segments found. Creating new log.
I20260928 21:09:57.686779 6087 log.cc:824] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:09:57.688467 6087 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: No bootstrap required, opened a new log
I20260928 21:09:57.688678 6087 ts_tablet_manager.cc:1397] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:09:57.689165 6086 raft_consensus.cc:348] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.689381 6086 raft_consensus.cc:374] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:57.689440 6086 raft_consensus.cc:729] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f875d8bcc9dd4f90927b23ee2e048246, State: Initialized, Role: FOLLOWER
I20260928 21:09:57.689890 6086 consensus_queue.cc:260] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.690590 6073 heartbeater.cc:503] Master 127.21.140.126:40549 was elected leader, sending a full tablet report...
I20260928 21:09:57.690907 6086 ts_tablet_manager.cc:1428] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:09:57.692003 6087 raft_consensus.cc:348] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.692286 6087 raft_consensus.cc:374] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:09:57.692346 6087 raft_consensus.cc:729] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5adf57374254ab3a2764b83c2823e9e, State: Initialized, Role: FOLLOWER
I20260928 21:09:57.692751 6087 consensus_queue.cc:260] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.693425 5940 heartbeater.cc:503] Master 127.21.140.126:40549 was elected leader, sending a full tablet report...
I20260928 21:09:57.694741 6087 ts_tablet_manager.cc:1428] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:09:57.762284 6090 raft_consensus.cc:482] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:09:57.762501 6090 raft_consensus.cc:504] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.763741 6090 leader_election.cc:288] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5adf57374254ab3a2764b83c2823e9e (127.21.140.66:46271), 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337)
I20260928 21:09:57.768323 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.763898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46271 (local address 127.21.140.67:50103)
0928 21:09:57.764046 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.764063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.764171 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.764517 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.764520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.764542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:57.764673 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.764680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.765624 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.765626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.766859 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.766869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.767791 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.767795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.767957 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.767987 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.768066 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.768173 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 21:09:57.768337 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.764057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50103 (local address 127.21.140.66:46271)
0928 21:09:57.764238 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.764245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.764261 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:09:57.764342 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.764345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.764395 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.764475 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:57.764791 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.765505 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.766989 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.767611 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.767735 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.767796 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.768137 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.768213 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 21:09:57.768635 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.764054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.21.140.67:40677)
0928 21:09:57.764533 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.764547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:09:57.764693 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:09:57.764991 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:09:57.764993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:09:57.765005 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:09:57.765141 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.765146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.766021 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.766025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.767204 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:09:57.767214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.768178 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.768181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:09:57.768280 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.768301 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:09:57.768380 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:09:57.768455 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":129,"threads_started":1}
I20260928 21:09:57.768895 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.764207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:40677 (local address 127.21.140.65:35337)
0928 21:09:57.764524 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.764529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.764677 (+ 148us) server_negotiation.cc:407] Connection header received
0928 21:09:57.764797 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.764800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.764864 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.764978 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:09:57.765249 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.765887 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.767309 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.768026 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.768171 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.768533 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.768585 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.768660 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260928 21:09:57.769136 5894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fd378576947a786eb1546e9c7b3f8" candidate_uuid: "f875d8bcc9dd4f90927b23ee2e048246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5adf57374254ab3a2764b83c2823e9e" is_pre_election: true
I20260928 21:09:57.769483 5894 raft_consensus.cc:2454] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f875d8bcc9dd4f90927b23ee2e048246 in term 0.
I20260928 21:09:57.769438 5755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fd378576947a786eb1546e9c7b3f8" candidate_uuid: "f875d8bcc9dd4f90927b23ee2e048246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9272954df65543b9bb73add09a102a0e" is_pre_election: true
I20260928 21:09:57.769717 5755 raft_consensus.cc:2454] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f875d8bcc9dd4f90927b23ee2e048246 in term 0.
I20260928 21:09:57.770081 5960 leader_election.cc:302] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5adf57374254ab3a2764b83c2823e9e, f875d8bcc9dd4f90927b23ee2e048246; no voters:
I20260928 21:09:57.770380 6090 raft_consensus.cc:2781] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:09:57.770468 6090 raft_consensus.cc:482] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:09:57.770511 6090 raft_consensus.cc:3037] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:57.771749 6090 raft_consensus.cc:504] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.772135 6090 leader_election.cc:288] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [CANDIDATE]: Term 1 election: Requested vote from peers e5adf57374254ab3a2764b83c2823e9e (127.21.140.66:46271), 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337)
I20260928 21:09:57.772502 5894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fd378576947a786eb1546e9c7b3f8" candidate_uuid: "f875d8bcc9dd4f90927b23ee2e048246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5adf57374254ab3a2764b83c2823e9e"
I20260928 21:09:57.772557 5755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fd378576947a786eb1546e9c7b3f8" candidate_uuid: "f875d8bcc9dd4f90927b23ee2e048246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9272954df65543b9bb73add09a102a0e"
I20260928 21:09:57.772650 5894 raft_consensus.cc:3037] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:57.772742 5755 raft_consensus.cc:3037] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:09:57.774009 5755 raft_consensus.cc:2454] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f875d8bcc9dd4f90927b23ee2e048246 in term 1.
I20260928 21:09:57.774009 5894 raft_consensus.cc:2454] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f875d8bcc9dd4f90927b23ee2e048246 in term 1.
I20260928 21:09:57.774339 5961 leader_election.cc:302] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9272954df65543b9bb73add09a102a0e, f875d8bcc9dd4f90927b23ee2e048246; no voters:
I20260928 21:09:57.774545 6090 raft_consensus.cc:2781] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:09:57.774770 6090 raft_consensus.cc:686] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 1 LEADER]: Becoming Leader. State: Replica: f875d8bcc9dd4f90927b23ee2e048246, State: Running, Role: LEADER
I20260928 21:09:57.775058 6090 consensus_queue.cc:237] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:09:57.778648 5610 catalog_manager.cc:5495] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 reported cstate change: term changed from 0 to 1, leader changed from <none> to f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "f875d8bcc9dd4f90927b23ee2e048246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } health_report { overall_health: UNKNOWN } } }
I20260928 21:09:57.794250 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.790496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.21.140.67:35327)
0928 21:09:57.790664 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 21:09:57.790668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:09:57.790683 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:09:57.790781 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:09:57.790783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:09:57.790835 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:09:57.790932 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:09:57.791235 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.791754 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.792714 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:09:57.792926 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:09:57.793008 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:09:57.793289 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:09:57.793715 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:09:57.793805 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:09:57.793911 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:09:57.793958 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 21:09:57.800660 5755 raft_consensus.cc:1261] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Refusing update from remote peer f875d8bcc9dd4f90927b23ee2e048246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:57.800662 5894 raft_consensus.cc:1261] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Refusing update from remote peer f875d8bcc9dd4f90927b23ee2e048246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:09:57.801386 6094 consensus_queue.cc:1035] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:57.801579 6090 consensus_queue.cc:1035] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:09:57.807683 6097 mvcc.cc:205] Tried to move back new op lower bound from 7334419651783335936 to 7334419651689312256. Current Snapshot: MvccSnapshot[applied={T|T < 7334419651783335936}]
I20260928 21:09:57.817181 5610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:57.817339 5610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:09:57.819787 5610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Sending DeleteTablet for 3 replicas of tablet a13fd378576947a786eb1546e9c7b3f8
I20260928 21:09:57.820464 5874 tablet_service.cc:1514] Processing DeleteTablet for tablet a13fd378576947a786eb1546e9c7b3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:57 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:46218
I20260928 21:09:57.820585 6007 tablet_service.cc:1514] Processing DeleteTablet for tablet a13fd378576947a786eb1546e9c7b3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:09:57 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:40264
I20260928 21:09:57.820760 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5650
I20260928 21:09:57.826238 6107 tablet_replica.cc:330] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: stopping tablet replica
I20260928 21:09:57.826236 6106 tablet_replica.cc:330] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: stopping tablet replica
I20260928 21:09:57.826649 6106 raft_consensus.cc:2229] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 1 LEADER]: Raft consensus shutting down.
W20260928 21:09:57.826843 5597 connection.cc:535] client connection to 127.21.140.65:35337 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:09:57.827001 5597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:09:57.827224 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5805
I20260928 21:09:57.826666 6107 raft_consensus.cc:2229] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:09:57.828651 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.828335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.0.0.1:53984)
0928 21:09:57.828502 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.828567 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260928 21:09:57.828883 5597 catalog_manager.cc:4584] TS 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a13fd378576947a786eb1546e9c7b3f8: Network error: Client connection negotiation failed: client connection to 127.21.140.65:35337: connect: Connection refused (error 111)
I20260928 21:09:57.829583 6106 raft_consensus.cc:2258] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:09:57.831317 6106 ts_tablet_manager.cc:1905] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:09:57.834556 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5943
I20260928 21:09:57.835345 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:09:57.834709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46271 (local address 127.0.0.1:46232)
0928 21:09:57.835214 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 21:09:57.835266 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
I20260928 21:09:57.839787 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 5578
I20260928 21:10:17.846012 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40549
--webserver_interface=127.21.140.126
--webserver_port=41441
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:17.988842 6109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:17.989251 6109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:17.989300 6109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:17.993914 6109 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:10:17.994055 6109 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:10:17.994073 6109 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:10:17.994088 6109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:17.994110 6109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:17.994129 6109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:17.994203 6109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:17.994235 6109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:18.000700 6109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=41441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:18.002442 6109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:18.003984 6109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:10:18.016077 6109 server_base.cc:1034] running on GCE node
W20260928 21:10:18.016014 6114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:18.016014 6115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:18.016017 6117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.017242 6109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:18.019155 6109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:18.020390 6109 hybrid_clock.cc:648] HybridClock initialized: now 1790629818020343 us; error 83 us; skew 500 ppm
Sep 28 21:10:18 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:18.032787 6109 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:10:18.034471 6109 webserver.cc:460] Webserver started at http://127.21.140.126:41441/ using document root <none> and password file <none>
I20260928 21:10:18.035364 6109 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:18.035487 6109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:18.041075 6109 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:18.044188 6124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:18.045949 6109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:10:18.046235 6109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "1506ac661a2e463dac392a2c6c36478b"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "ace067dad802d98f5ec8fdce5b776334"
server_key_iv: "f2454b64c1a1768b1937a209f42a776d"
server_key_version: "encryptionkey@0"
I20260928 21:10:18.047060 6109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:18.074458 6109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:18.078812 6109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:18.079101 6109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:18.088879 6109 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40549
I20260928 21:10:18.089257 6177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40549 every 8 connection(s)
I20260928 21:10:18.090513 6109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:18.096589 6178 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:10:18.097069 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6109
I20260928 21:10:18.097666 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:35337
--local_ip_for_outbound_sockets=127.21.140.65
--tserver_master_addrs=127.21.140.126:40549
--webserver_port=46611
--webserver_interface=127.21.140.65
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:10:18.100025 6178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap starting.
I20260928 21:10:18.122620 6178 log.cc:824] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:18.136487 6178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:10:18.137023 6178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: Bootstrap complete.
I20260928 21:10:18.142020 6178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:10:18.142653 6178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1506ac661a2e463dac392a2c6c36478b, State: Initialized, Role: FOLLOWER
I20260928 21:10:18.143401 6178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:10:18.143605 6178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:18.143702 6178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:18.143846 6178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:10:18.146695 6178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:10:18.147221 6178 leader_election.cc:302] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1506ac661a2e463dac392a2c6c36478b; no voters:
I20260928 21:10:18.147713 6178 leader_election.cc:288] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:10:18.147850 6182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:10:18.148113 6182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [term 3 LEADER]: Becoming Leader. State: Replica: 1506ac661a2e463dac392a2c6c36478b, State: Running, Role: LEADER
I20260928 21:10:18.148552 6182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } }
I20260928 21:10:18.149529 6178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:18.150861 6183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1506ac661a2e463dac392a2c6c36478b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } } }
I20260928 21:10:18.151055 6183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:18.151674 6184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1506ac661a2e463dac392a2c6c36478b. Latest consensus state: current_term: 3 leader_uuid: "1506ac661a2e463dac392a2c6c36478b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1506ac661a2e463dac392a2c6c36478b" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40549 } } }
I20260928 21:10:18.151861 6184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:18.156718 6198 catalog_manager.cc:1259] Loaded cluster ID: 81f8bcf1ac9b4955ac8fc1d941a76312
I20260928 21:10:18.156889 6198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: loading cluster ID for follower catalog manager: success
I20260928 21:10:18.159488 6198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1506ac661a2e463dac392a2c6c36478b: acquiring CA information for follower catalog manager: success
F20260928 21:10:18.161089 6198 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5f2797f9ad google::LogMessage::Fail() at ??:0
@ 0x7f5f2798288a google::LogMessage::SendToLog() at ??:0
@ 0x7f5f2797f4de google::LogMessage::Flush() at ??:0
@ 0x7f5f27980a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5f2a64318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5f322e4989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f5f322c1023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f5f322c154b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f5f322ba652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f5f322b99e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f5f32300541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5f29bc2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5f2b2066db start_thread at ??:0
@ 0x7f5f2647d61f clone at ??:0
W20260928 21:10:18.291137 6180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:18.291725 6180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:18.291843 6180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:18.299232 6180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:18.299366 6180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:18.299474 6180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:18.309906 6180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:35337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=46611
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:18.311969 6180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:18.313853 6180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:18.329911 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:18.329854 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.329879 6180 server_base.cc:1034] running on GCE node
W20260928 21:10:18.330193 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.331004 6180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:18.332297 6180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:18.333684 6180 hybrid_clock.cc:648] HybridClock initialized: now 1790629818333613 us; error 92 us; skew 500 ppm
Sep 28 21:10:18 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:18.339074 6180 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:10:18.341168 6180 webserver.cc:460] Webserver started at http://127.21.140.65:46611/ using document root <none> and password file <none>
I20260928 21:10:18.342316 6180 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:18.342465 6180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:18.349860 6180 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 21:10:18.354176 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:18.356539 6180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:10:18.356771 6180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "9272954df65543b9bb73add09a102a0e"
format_stamp: "Formatted at 2026-09-28 21:09:35 on dist-test-slave-d5wc"
server_key: "4a43be82696dbf622e2e2f156ff99dd2"
server_key_iv: "2f0c4a7d61e6a364f97d9478cb00e45d"
server_key_version: "encryptionkey@0"
I20260928 21:10:18.357846 6180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:18.384454 6180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:18.390158 6180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:18.390479 6180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:18.391525 6180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:18.393508 6235 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:10:18.399730 6180 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:10:18.399885 6180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 21:10:18.399983 6180 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:10:18.403368 6180 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:10:18.403491 6180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260928 21:10:18.404042 6235 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Bootstrap starting.
I20260928 21:10:18.420174 6180 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:35337
I20260928 21:10:18.420849 6343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:35337 every 8 connection(s)
I20260928 21:10:18.422124 6180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:18.430094 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6180
I20260928 21:10:18.430711 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:46271
--local_ip_for_outbound_sockets=127.21.140.66
--tserver_master_addrs=127.21.140.126:40549
--webserver_port=32983
--webserver_interface=127.21.140.66
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:10:18.433073 6235 log.cc:824] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:18.443869 6235 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:10:18.444464 6235 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Bootstrap complete.
I20260928 21:10:18.445915 6235 ts_tablet_manager.cc:1397] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.009s
I20260928 21:10:18.449247 6235 raft_consensus.cc:348] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:18.449870 6235 raft_consensus.cc:729] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9272954df65543b9bb73add09a102a0e, State: Initialized, Role: FOLLOWER
I20260928 21:10:18.450592 6235 consensus_queue.cc:260] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:18.451920 6235 ts_tablet_manager.cc:1428] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 21:10:18.478348 6346 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:10:18.425728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:41937)
0928 21:10:18.426721 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.426813 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:18.428502 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:18.477181 (+ 48679us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":325,"threads_started":1}
W20260928 21:10:18.479101 6344 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:40549 (0 consecutive failures): Network error: Failed to ping master at 127.21.140.126:40549: Client connection negotiation failed: client connection to 127.21.140.126:40549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:10:18.480159 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.479825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:49633)
0928 21:10:18.479998 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.480082 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 21:10:18.481084 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.480758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:33329)
0928 21:10:18.480928 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.480994 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 21:10:18.481316 6344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:10:18.589270 6347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:18.589681 6347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:18.589735 6347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:18.594236 6347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:18.594329 6347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:18.594399 6347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:18.600639 6347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:46271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=32983
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:18.602173 6347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:18.603467 6347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:18.616371 6355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:18.616364 6357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.616425 6347 server_base.cc:1034] running on GCE node
W20260928 21:10:18.616364 6354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.617430 6347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:18.618475 6347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:18.619726 6347 hybrid_clock.cc:648] HybridClock initialized: now 1790629818619699 us; error 45 us; skew 500 ppm
Sep 28 21:10:18 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:18.623520 6347 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:10:18.624789 6347 webserver.cc:460] Webserver started at http://127.21.140.66:32983/ using document root <none> and password file <none>
I20260928 21:10:18.625569 6347 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:18.625656 6347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:18.631488 6347 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:10:18.634595 6364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:18.636624 6347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:10:18.636844 6347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "e5adf57374254ab3a2764b83c2823e9e"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "72b0d0c2dd6747942c43a2e85e2ca5a9"
server_key_iv: "4226cb81a27ade996fc24b7241c374d6"
server_key_version: "encryptionkey@0"
I20260928 21:10:18.637696 6347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:18.664204 6347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:18.668843 6347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:18.669123 6347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:18.670028 6347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:18.671486 6371 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:10:18.677735 6347 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:10:18.677871 6347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 21:10:18.678035 6347 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:10:18.680595 6347 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:10:18.680671 6347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260928 21:10:18.681066 6371 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Bootstrap starting.
I20260928 21:10:18.694388 6347 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:46271
I20260928 21:10:18.694499 6479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:46271 every 8 connection(s)
I20260928 21:10:18.695863 6347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:18.702044 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.699141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:43129)
0928 21:10:18.699980 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.700182 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":208,"threads_started":1}
W20260928 21:10:18.702795 6480 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:40549 (0 consecutive failures): Network error: Failed to ping master at 127.21.140.126:40549: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
I20260928 21:10:18.703894 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.703514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:53603)
0928 21:10:18.703699 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.703784 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 21:10:18.704803 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.704511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:59565)
0928 21:10:18.704652 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.704716 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 21:10:18.705068 6480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:10:18.705410 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6347
I20260928 21:10:18.706050 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:35327
--local_ip_for_outbound_sockets=127.21.140.67
--tserver_master_addrs=127.21.140.126:40549
--webserver_port=44215
--webserver_interface=127.21.140.67
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:10:18.706897 6371 log.cc:824] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:18.718447 6371 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:10:18.719017 6371 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Bootstrap complete.
I20260928 21:10:18.720525 6371 ts_tablet_manager.cc:1397] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.004s
I20260928 21:10:18.724452 6371 raft_consensus.cc:348] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:18.725129 6371 raft_consensus.cc:729] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5adf57374254ab3a2764b83c2823e9e, State: Initialized, Role: FOLLOWER
I20260928 21:10:18.725849 6371 consensus_queue.cc:260] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:18.727336 6371 ts_tablet_manager.cc:1428] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e: Time spent starting tablet: real 0.007s user 0.001s sys 0.008s
I20260928 21:10:18.727814 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.727439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:47859)
0928 21:10:18.727613 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.727717 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 21:10:18.728794 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.728538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:56867)
0928 21:10:18.728674 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.728726 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 21:10:18.857213 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:18.857523 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:18.857571 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:18.861703 6484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:18.861765 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:18.861866 6484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:18.867712 6484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:35327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=44215
--tserver_master_addrs=127.21.140.126:40549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:18.868937 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:18.870401 6484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:18.881639 6490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:18.881665 6493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.881726 6484 server_base.cc:1034] running on GCE node
W20260928 21:10:18.881666 6491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:18.882535 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:18.883198 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:18.884378 6484 hybrid_clock.cc:648] HybridClock initialized: now 1790629818884341 us; error 49 us; skew 500 ppm
Sep 28 21:10:18 dist-test-slave-d5wc krb5kdc[5026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:18.887370 6484 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:10:18.888392 6484 webserver.cc:460] Webserver started at http://127.21.140.67:44215/ using document root <none> and password file <none>
I20260928 21:10:18.889043 6484 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:18.889108 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:18.893414 6484 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:10:18.895882 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:18.897311 6484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:10:18.897483 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "f875d8bcc9dd4f90927b23ee2e048246"
format_stamp: "Formatted at 2026-09-28 21:09:36 on dist-test-slave-d5wc"
server_key: "70dcfa2883c7584be7955559149e88a4"
server_key_iv: "29f63dee6ed3e7586a8120b2410a4397"
server_key_version: "encryptionkey@0"
I20260928 21:10:18.898129 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:18.931442 6484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:18.935127 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:18.935346 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:18.936012 6484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:18.937213 6508 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 21:10:18.941587 6508 ts_tablet_manager.cc:1852] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 21:10:18.942124 6508 ts_tablet_manager.cc:1905] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:18.946468 6508 ts_tablet_manager.cc:1918] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:18.946774 6508 log.cc:1196] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/a13fd378576947a786eb1546e9c7b3f8
I20260928 21:10:18.947264 6508 ts_tablet_manager.cc:1939] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Deleting consensus metadata
I20260928 21:10:18.947634 6484 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 21:10:18.947708 6484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.000s
I20260928 21:10:18.947815 6484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:18.947866 6484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:18.957391 6484 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35327
I20260928 21:10:18.957432 6615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35327 every 8 connection(s)
I20260928 21:10:18.958407 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:10:18.962033 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.960651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:50773)
0928 21:10:18.961133 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.961242 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
W20260928 21:10:18.962460 6616 heartbeater.cc:644] Failed to heartbeat to 127.21.140.126:40549 (0 consecutive failures): Network error: Failed to ping master at 127.21.140.126:40549: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
I20260928 21:10:18.963073 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.962854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:41433)
0928 21:10:18.962968 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.963024 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 21:10:18.963722 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:18.963496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:36035)
0928 21:10:18.963602 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:18.963677 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:10:18.963934 6616 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:10:18.966922 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6484
I20260928 21:10:19.482628 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.482114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:49861)
0928 21:10:19.482411 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:19.482486 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
I20260928 21:10:19.730300 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.729716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:51903)
0928 21:10:19.730115 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:19.730197 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":86,"threads_started":1}
I20260928 21:10:19.818913 6622 raft_consensus.cc:482] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:19.819159 6622 raft_consensus.cc:504] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:19.820390 6622 leader_election.cc:288] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e5adf57374254ab3a2764b83c2823e9e (127.21.140.66:46271), f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327)
Sep 28 21:10:19 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.66@KRBTEST.COM
Sep 28 21:10:19 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.67@KRBTEST.COM
I20260928 21:10:19.834923 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.820651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35327 (local address 127.21.140.65:41881)
0928 21:10:19.820914 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:19.820931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:19.821070 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:19.822716 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:19.822719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:19.823302 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:19.823785 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:19.823800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.825058 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.825062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:19.826048 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:19.826057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.826776 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.826780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:19.826947 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:19.827670 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:19.827682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:19.829693 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:19.832043 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:19.832050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:19.832066 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:19.832551 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:19.832944 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:19.832949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:19.832951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:19.833179 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:19.833849 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:19.833856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:19.834149 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 21:10:19.834333 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 21:10:19.834708 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":102,"spinlock_wait_cycles":27264}
I20260928 21:10:19.834932 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.820864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:47147 (local address 127.21.140.66:46271)
0928 21:10:19.821369 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:19.821381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:19.821415 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:10:19.821710 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:19.821716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:19.822265 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:19.822613 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:19.823990 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.825003 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.826182 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.826740 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.826924 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:19.829581 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:19.829587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:19.829600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:19.829628 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:19.831862 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:19.832711 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:19.832715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:19.832719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:19.832885 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:19.833277 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:19.833282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:19.833284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:19.833739 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:19.833890 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:19.834470 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 21:10:19.834744 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":80,"threads_started":1}
I20260928 21:10:19.834962 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.820812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:41881 (local address 127.21.140.67:35327)
0928 21:10:19.821216 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:19.821228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:19.821264 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:10:19.822202 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:19.822210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:19.822506 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:19.822772 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:19.823951 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.824907 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.826163 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.826647 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.826824 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:19.829888 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:19.829895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:19.829910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:19.829945 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:19.831893 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:19.832687 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:19.832693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:19.832698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:19.832851 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:19.833325 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:19.833327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:19.833329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:19.833721 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:19.833994 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:19.834488 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 21:10:19.834774 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":90,"threads_started":1}
I20260928 21:10:19.834966 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.820720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:46271 (local address 127.21.140.65:47147)
0928 21:10:19.821097 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:19.821113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:19.821214 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:19.822666 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:19.822673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:19.823303 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:19.823786 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:19.823800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.825155 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.825160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:19.826042 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:19.826060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:19.826876 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:19.826880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:19.826980 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:19.827632 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:19.827649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:19.829348 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:19.832043 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:19.832050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:19.832066 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:19.832578 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:19.833005 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:19.833009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:19.833015 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:19.833171 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:19.834006 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:19.834013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:19.834147 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 21:10:19.834338 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 21:10:19.834792 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":220,"spinlock_wait_cycles":40960,"thread_start_us":91,"threads_started":1}
I20260928 21:10:19.837772 6434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fd378576947a786eb1546e9c7b3f8" candidate_uuid: "9272954df65543b9bb73add09a102a0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e5adf57374254ab3a2764b83c2823e9e" is_pre_election: true
W20260928 21:10:19.838056 6231 leader_election.cc:341] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327): Not found: Tablet not found: a13fd378576947a786eb1546e9c7b3f8
I20260928 21:10:19.838155 6434 raft_consensus.cc:2454] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9272954df65543b9bb73add09a102a0e in term 1.
I20260928 21:10:19.838881 6230 leader_election.cc:302] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9272954df65543b9bb73add09a102a0e, e5adf57374254ab3a2764b83c2823e9e; no voters: f875d8bcc9dd4f90927b23ee2e048246
I20260928 21:10:19.839134 6622 raft_consensus.cc:2781] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:10:19.839226 6622 raft_consensus.cc:482] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:19.839284 6622 raft_consensus.cc:3037] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:10:19.841450 6622 raft_consensus.cc:504] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:19.841827 6622 leader_election.cc:288] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [CANDIDATE]: Term 2 election: Requested vote from peers e5adf57374254ab3a2764b83c2823e9e (127.21.140.66:46271), f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327)
W20260928 21:10:19.842468 6231 leader_election.cc:341] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327): Not found: Tablet not found: a13fd378576947a786eb1546e9c7b3f8
I20260928 21:10:19.842448 6434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fd378576947a786eb1546e9c7b3f8" candidate_uuid: "9272954df65543b9bb73add09a102a0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e5adf57374254ab3a2764b83c2823e9e"
I20260928 21:10:19.842615 6434 raft_consensus.cc:3037] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:10:19.844820 6434 raft_consensus.cc:2454] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9272954df65543b9bb73add09a102a0e in term 2.
I20260928 21:10:19.845140 6230 leader_election.cc:302] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9272954df65543b9bb73add09a102a0e, e5adf57374254ab3a2764b83c2823e9e; no voters: f875d8bcc9dd4f90927b23ee2e048246
I20260928 21:10:19.845309 6622 raft_consensus.cc:2781] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:10:19.845539 6622 raft_consensus.cc:686] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [term 2 LEADER]: Becoming Leader. State: Replica: 9272954df65543b9bb73add09a102a0e, State: Running, Role: LEADER
I20260928 21:10:19.845923 6622 consensus_queue.cc:237] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:19.846272 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.846014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:41857)
0928 21:10:19.846136 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:19.846193 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 21:10:19.964915 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:19.964474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:56843)
0928 21:10:19.964766 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:19.964837 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":133,"threads_started":1}
W20260928 21:10:20.306157 6231 consensus_peers.cc:487] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e -> Peer f875d8bcc9dd4f90927b23ee2e048246 (127.21.140.67:35327): Couldn't send request to peer f875d8bcc9dd4f90927b23ee2e048246. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a13fd378576947a786eb1546e9c7b3f8. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:10:20.334203 6434 raft_consensus.cc:1261] T a13fd378576947a786eb1546e9c7b3f8 P e5adf57374254ab3a2764b83c2823e9e [term 2 FOLLOWER]: Refusing update from remote peer 9272954df65543b9bb73add09a102a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 21:10:20.335193 6622 consensus_queue.cc:1035] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 21:10:20.337394 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.337062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:37905)
0928 21:10:20.337262 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:20.337313 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260928 21:10:20.339630 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.339379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:33335)
0928 21:10:20.339488 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:20.339547 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:10:20.340106 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.339886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:54607)
0928 21:10:20.339996 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:20.340041 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:10:20.755995 6634 ts_tablet_manager.cc:927] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Initiating tablet copy from peer 9272954df65543b9bb73add09a102a0e (127.21.140.65:35337)
I20260928 21:10:20.756628 6634 tablet_copy_client.cc:316] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: tablet copy: Beginning tablet copy session from remote peer at address 127.21.140.65:35337
Sep 28 21:10:20 dist-test-slave-d5wc krb5kdc[5026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.65@KRBTEST.COM
I20260928 21:10:20.767990 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.757202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35337 (local address 127.21.140.67:51259)
0928 21:10:20.757485 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:20.757513 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:20.757702 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:20.758266 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:20.758272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:20.758725 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:20.758839 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:20.758845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:20.759871 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:20.759878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:20.760936 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:20.760947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:20.761656 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:20.761659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:20.761765 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:20.762458 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:20.762473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:20.764232 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:20.766246 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:20.766255 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:20.766260 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:20.766566 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:20.766893 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:20.766895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:20.766896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:20.766976 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:20.767433 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:20.767438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:20.767545 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:10:20.767667 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:10:20.767844 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":128,"threads_started":1}
I20260928 21:10:20.768036 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.757410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:51259 (local address 127.21.140.65:35337)
0928 21:10:20.757775 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:20.757790 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:20.757816 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:10:20.757867 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:20.757872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:20.758115 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:20.758266 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:20.758970 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:20.759756 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:20.761064 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:20.761530 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:20.761652 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:20.764385 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:20.764395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:20.764411 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:20.764446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:20.766145 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:20.766714 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:20.766723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:20.766728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:20.766809 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:20.767054 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:20.767057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:20.767058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:20.767335 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:20.767441 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:20.767748 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:10:20.767906 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260928 21:10:20.769496 6318 tablet_copy_service.cc:140] P 9272954df65543b9bb73add09a102a0e: Received BeginTabletCopySession request for tablet a13fd378576947a786eb1546e9c7b3f8 from peer f875d8bcc9dd4f90927b23ee2e048246 ({username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:51259)
I20260928 21:10:20.769714 6318 tablet_copy_service.cc:161] P 9272954df65543b9bb73add09a102a0e: Beginning new tablet copy session on tablet a13fd378576947a786eb1546e9c7b3f8 from peer f875d8bcc9dd4f90927b23ee2e048246 at {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:51259: session id = f875d8bcc9dd4f90927b23ee2e048246-a13fd378576947a786eb1546e9c7b3f8
I20260928 21:10:20.770964 6318 tablet_copy_source_session.cc:205] T a13fd378576947a786eb1546e9c7b3f8 P 9272954df65543b9bb73add09a102a0e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:10:20.773773 6634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fd378576947a786eb1546e9c7b3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:20.778174 6634 tablet_copy_client.cc:791] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: tablet copy: Starting download of 0 data blocks...
I20260928 21:10:20.778369 6634 tablet_copy_client.cc:655] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: tablet copy: Starting download of 1 WAL segments...
I20260928 21:10:20.780349 6634 tablet_copy_client.cc:531] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:10:20.782336 6634 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Bootstrap starting.
I20260928 21:10:20.788358 6634 log.cc:824] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:20.795449 6634 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:10:20.795773 6634 tablet_bootstrap.cc:492] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Bootstrap complete.
I20260928 21:10:20.796109 6634 ts_tablet_manager.cc:1397] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 21:10:20.798862 6634 raft_consensus.cc:348] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:20.799136 6634 raft_consensus.cc:729] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f875d8bcc9dd4f90927b23ee2e048246, State: Initialized, Role: FOLLOWER
I20260928 21:10:20.799606 6634 consensus_queue.cc:260] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5adf57374254ab3a2764b83c2823e9e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 46271 } } peers { permanent_uuid: "f875d8bcc9dd4f90927b23ee2e048246" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35327 } } peers { permanent_uuid: "9272954df65543b9bb73add09a102a0e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35337 } }
I20260928 21:10:20.800643 6634 ts_tablet_manager.cc:1428] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:20.800859 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.800584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:60119)
0928 21:10:20.800724 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:20.800792 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 21:10:20.801429 6318 tablet_copy_service.cc:342] P 9272954df65543b9bb73add09a102a0e: Request end of tablet copy session f875d8bcc9dd4f90927b23ee2e048246-a13fd378576947a786eb1546e9c7b3f8 received from {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:51259
I20260928 21:10:20.801551 6318 tablet_copy_service.cc:433] P 9272954df65543b9bb73add09a102a0e: ending tablet copy session f875d8bcc9dd4f90927b23ee2e048246-a13fd378576947a786eb1546e9c7b3f8 on tablet a13fd378576947a786eb1546e9c7b3f8 with peer f875d8bcc9dd4f90927b23ee2e048246
I20260928 21:10:20.801774 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:20.801459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:48229)
0928 21:10:20.801655 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:20.801719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 21:10:21.132783 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:21.132357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:52235)
0928 21:10:21.132599 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:21.132662 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":54,"threads_started":1}
I20260928 21:10:21.132858 6570 raft_consensus.cc:1203] T a13fd378576947a786eb1546e9c7b3f8 P f875d8bcc9dd4f90927b23ee2e048246 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 21:10:21.134276 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:21.133862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:43027)
0928 21:10:21.134073 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:21.134166 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 21:10:21.341614 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:21.341075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:41047)
0928 21:10:21.341407 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:21.341482 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260928 21:10:22.134272 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:22.133688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:41857)
0928 21:10:22.134053 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:22.134150 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260928 21:10:22.135485 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:22.135079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:35881)
0928 21:10:22.135333 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:22.135397 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260928 21:10:22.343122 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:22.342538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:41075)
0928 21:10:22.342877 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:22.342937 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":80,"threads_started":1}
I20260928 21:10:23.135471 6648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:23.135052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:33121)
0928 21:10:23.135308 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:23.135374 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260928 21:10:23.136659 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:23.136240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:35545)
0928 21:10:23.136511 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:23.136573 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":64,"threads_started":1}
I20260928 21:10:23.344551 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:23.344023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:42585)
0928 21:10:23.344383 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:23.344450 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260928 21:10:24.137467 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:24.136809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:33925)
0928 21:10:24.137214 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:24.137312 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":140,"threads_started":1}
I20260928 21:10:24.138394 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:24.138138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:57087)
0928 21:10:24.138258 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:24.138321 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:10:24.138407 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:24.137882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:36517)
0928 21:10:24.138239 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:24.138302 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":174,"threads_started":1}
I20260928 21:10:24.346391 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:24.345784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:56701)
0928 21:10:24.346182 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:24.346255 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260928 21:10:25.139708 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:25.139235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:37035)
0928 21:10:25.139527 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:25.139590 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
I20260928 21:10:25.139729 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:25.139266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:52887)
0928 21:10:25.139553 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:25.139629 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":116,"threads_started":1}
I20260928 21:10:25.347709 6656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:25.347239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:48887)
0928 21:10:25.347535 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:25.347599 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
I20260928 21:10:26.140990 6657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:26.140570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:34959)
0928 21:10:26.140836 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:26.140894 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260928 21:10:26.141106 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:26.140667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:38353)
0928 21:10:26.140920 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:26.140983 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":113,"threads_started":1}
I20260928 21:10:26.349078 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:26.348598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:51155)
0928 21:10:26.348887 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:26.348948 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260928 21:10:27.142649 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:27.142053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:48889)
0928 21:10:27.142398 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:27.142488 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260928 21:10:27.142668 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:27.142075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:58209)
0928 21:10:27.142502 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:27.142567 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":100,"threads_started":1}
I20260928 21:10:27.350872 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:27.350286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:59705)
0928 21:10:27.350651 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:27.350730 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260928 21:10:28.144270 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:28.143712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:44259)
0928 21:10:28.144109 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:28.144168 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":120,"threads_started":1}
I20260928 21:10:28.144258 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:28.143712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:57089)
0928 21:10:28.144048 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:28.144126 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":119,"threads_started":1}
I20260928 21:10:28.352406 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:28.351818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:48109)
0928 21:10:28.352181 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:28.352251 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260928 21:10:29.145995 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:29.145371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:33015)
0928 21:10:29.145798 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:29.145863 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":113,"threads_started":1}
I20260928 21:10:29.145973 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:29.145429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:41329)
0928 21:10:29.145746 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:29.145820 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260928 21:10:29.354386 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:29.353684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:45369)
0928 21:10:29.354160 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:29.354248 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":116,"threads_started":1}
I20260928 21:10:30.147616 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:30.147068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:60667)
0928 21:10:30.147408 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:30.147487 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
I20260928 21:10:30.147683 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:30.147068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:46959)
0928 21:10:30.147518 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:30.147580 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":120,"threads_started":1}
I20260928 21:10:30.356194 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:30.355560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:34805)
0928 21:10:30.355959 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:30.356042 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260928 21:10:31.149289 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:31.148728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:54429)
0928 21:10:31.149096 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:31.149163 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
I20260928 21:10:31.149294 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:31.148730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:47221)
0928 21:10:31.149082 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:31.149165 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260928 21:10:31.358047 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:31.357410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:39507)
0928 21:10:31.357759 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:31.357833 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260928 21:10:32.151170 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:32.150572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:58627)
0928 21:10:32.150913 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:32.151006 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":123,"threads_started":1}
I20260928 21:10:32.151212 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:32.150577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:33183)
0928 21:10:32.151022 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:32.151085 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":124,"threads_started":1}
I20260928 21:10:32.359663 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:32.359055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:54053)
0928 21:10:32.359432 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:32.359516 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260928 21:10:33.152988 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:33.152358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.67:39463)
0928 21:10:33.152708 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:33.152859 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":135,"threads_started":1}
I20260928 21:10:33.153087 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:33.152638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.65:60357)
0928 21:10:33.152903 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:33.152967 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260928 21:10:33.361438 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:33.360825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:40549 (local address 127.21.140.66:52497)
0928 21:10:33.361196 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:33.361292 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.126:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
I20260928 21:10:33.974152 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 6180
I20260928 21:10:33.981616 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 6347
I20260928 21:10:33.989154 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 6484
2026-09-28T21:10:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59781 ms)
[----------] 30 tests from SecurityITest (250516 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:34 dist-test-slave-d5wc krb5kdc[6683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:34 dist-test-slave-d5wc krb5kdc[6683](info): set up 2 sockets
Sep 28 21:10:34 dist-test-slave-d5wc krb5kdc[6683](info): commencing operation
krb5kdc: starting...
W20260928 21:10:35.060159 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:10:35.210093 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:45233
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:45233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:35.333282 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:35.333626 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:35.333704 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:35.337978 6696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:35.338071 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:35.338109 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:35.338168 6696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:35.338223 6696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:35.343750 6696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:45233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:35.344885 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:35.345944 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:35.355446 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:35.355436 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:35.355536 6696 server_base.cc:1034] running on GCE node
W20260928 21:10:35.355441 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:35.356323 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:35.357434 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:35.358639 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790629835358614 us; error 51 us; skew 500 ppm
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:35.361608 6696 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:10:35.362639 6696 webserver.cc:460] Webserver started at http://127.21.140.126:38587/ using document root <none> and password file <none>
I20260928 21:10:35.363241 6696 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:35.363340 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:35.363639 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:35.366178 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "2b6010bffd92439ca6504448ec2b9528"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "9fcc556adc3234040350d2e0699f1769"
server_key_iv: "b8ac6a3b1a28730c0ca5ac379f4480ee"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.366853 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "2b6010bffd92439ca6504448ec2b9528"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "9fcc556adc3234040350d2e0699f1769"
server_key_iv: "b8ac6a3b1a28730c0ca5ac379f4480ee"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.371090 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:10:35.373642 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.375025 6696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:10:35.375182 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "2b6010bffd92439ca6504448ec2b9528"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "9fcc556adc3234040350d2e0699f1769"
server_key_iv: "b8ac6a3b1a28730c0ca5ac379f4480ee"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.375311 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:35.398283 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:35.402241 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:35.402415 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:35.409502 6696 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:45233
I20260928 21:10:35.409634 6763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:45233 every 8 connection(s)
I20260928 21:10:35.410574 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:35.413506 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:35.418917 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6696
I20260928 21:10:35.419078 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:35.419395 22065 external_mini_cluster.cc:1428] Setting key b5e67f40f6181e2e297af8ca43b53d43
I20260928 21:10:35.419773 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: Bootstrap starting.
I20260928 21:10:35.422435 6764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:35.423414 6764 log.cc:824] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:35.425664 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: No bootstrap required, opened a new log
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:35.430431 6764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6010bffd92439ca6504448ec2b9528" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45233 } }
I20260928 21:10:35.430678 6764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:35.430776 6764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6010bffd92439ca6504448ec2b9528, State: Initialized, Role: FOLLOWER
I20260928 21:10:35.431216 6764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6010bffd92439ca6504448ec2b9528" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45233 } }
I20260928 21:10:35.431375 6764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:35.431475 6764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:35.431597 6764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:35.432863 6764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6010bffd92439ca6504448ec2b9528" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45233 } }
I20260928 21:10:35.433221 6764 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b6010bffd92439ca6504448ec2b9528; no voters:
I20260928 21:10:35.433580 6764 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:35.433658 6769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:35.433861 6769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [term 1 LEADER]: Becoming Leader. State: Replica: 2b6010bffd92439ca6504448ec2b9528, State: Running, Role: LEADER
I20260928 21:10:35.434274 6769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6010bffd92439ca6504448ec2b9528" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45233 } }
I20260928 21:10:35.435002 6764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:35.435375 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:35.420874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.21.140.126:45233)
0928 21:10:35.421320 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:35.421337 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:35.421373 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:10:35.422102 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:35.422122 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:35.422452 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:35.422790 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:35.423736 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.424710 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.425878 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.426440 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.426625 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:35.429691 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:35.429697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:35.429711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:35.429739 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:35.431797 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:35.432293 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:35.432298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:35.432303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:35.432418 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:35.432737 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:35.432740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:35.432741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:35.433265 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:35.433479 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:35.434258 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 21:10:35.434563 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":161,"threads_started":1}
I20260928 21:10:35.436326 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6010bffd92439ca6504448ec2b9528. Latest consensus state: current_term: 1 leader_uuid: "2b6010bffd92439ca6504448ec2b9528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6010bffd92439ca6504448ec2b9528" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45233 } } }
I20260928 21:10:35.436461 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:35.436816 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b6010bffd92439ca6504448ec2b9528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6010bffd92439ca6504448ec2b9528" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45233 } } }
I20260928 21:10:35.436903 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528 [sys.catalog]: This master's current role is: LEADER
W20260928 21:10:35.440389 6785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:10:35.440483 6785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:10:35.440582 6779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:35.441385 6779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:35.447463 6779 catalog_manager.cc:1347] Generated new cluster ID: 418a26babdb2424888a57fe0c5d61681
I20260928 21:10:35.447553 6779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:35.464080 6779 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:35.465405 6779 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:35.476752 6779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: Generated new TSK 0
I20260928 21:10:35.477744 6779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:10:35.507665 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:45233
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:35.620486 6791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:35.620754 6791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:35.620797 6791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:35.624959 6791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:35.625030 6791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:35.625133 6791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:35.630582 6791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:45233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:35.631685 6791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:35.632815 6791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:35.642859 6797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:35.642874 6796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:35.642930 6791 server_base.cc:1034] running on GCE node
W20260928 21:10:35.642907 6799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:35.643595 6791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:35.644225 6791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:35.645394 6791 hybrid_clock.cc:648] HybridClock initialized: now 1790629835645358 us; error 48 us; skew 500 ppm
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:35.648325 6791 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:10:35.649220 6791 webserver.cc:460] Webserver started at http://127.21.140.65:46309/ using document root <none> and password file <none>
I20260928 21:10:35.649780 6791 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:35.649842 6791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:35.650116 6791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:35.652127 6791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "fc32568606934360b531ca0d573b229b"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "db160a2ef2de70328161370dff4c5932"
server_key_iv: "b01af9eab66a4c56624b074416f66823"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.652719 6791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "fc32568606934360b531ca0d573b229b"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "db160a2ef2de70328161370dff4c5932"
server_key_iv: "b01af9eab66a4c56624b074416f66823"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.656728 6791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:35.659451 6806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.660799 6791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:10:35.660949 6791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "fc32568606934360b531ca0d573b229b"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "db160a2ef2de70328161370dff4c5932"
server_key_iv: "b01af9eab66a4c56624b074416f66823"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.661080 6791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:35.685434 6791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:35.689004 6791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:35.689246 6791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:35.689908 6791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:35.690830 6791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:35.690907 6791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.690986 6791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:35.691042 6791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.701285 6791 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:39051
I20260928 21:10:35.701308 6919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:39051 every 8 connection(s)
I20260928 21:10:35.702312 6791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:35.704768 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6791
I20260928 21:10:35.704918 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:10:35.705236 22065 external_mini_cluster.cc:1428] Setting key f13c2004d8f45a18ab4b1d27d5667318
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:10:35.718276 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:35.704423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:43507 (local address 127.21.140.126:45233)
0928 21:10:35.704583 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:35.704588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:35.706077 (+ 1489us) server_negotiation.cc:407] Connection header received
0928 21:10:35.707825 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:35.707837 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:35.707896 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:35.708005 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:35.709630 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.710240 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.711445 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.711698 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.711812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:35.714354 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:35.714357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:35.714360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:35.714384 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:35.715972 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:35.716593 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:35.716596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:35.716598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:35.716684 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:35.717054 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:35.717058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:35.717059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:35.717247 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:35.717373 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:35.717935 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 21:10:35.718119 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:10:35.719137 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:35.704721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:45233 (local address 127.21.140.65:43507)
0928 21:10:35.705848 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:35.705905 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:35.707327 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:35.708187 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:35.708195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:35.708764 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:35.709480 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:35.709503 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.710367 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.710370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:35.711309 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:35.711318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.711854 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.711858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:35.712032 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:35.712648 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:35.712664 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:35.714172 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:35.716099 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:35.716105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:35.716120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:35.716482 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:35.716788 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:35.716791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:35.716793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:35.716940 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:35.717373 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:35.717380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:35.717640 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 21:10:35.717844 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 21:10:35.718256 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":165,"threads_started":1}
I20260928 21:10:35.720642 6920 heartbeater.cc:347] Connected to a master server at 127.21.140.126:45233
I20260928 21:10:35.720917 6920 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:35.721588 6920 heartbeater.cc:511] Master 127.21.140.126:45233 requested a full tablet report, sending...
I20260928 21:10:35.723258 6728 ts_manager.cc:194] Registered new tserver with Master: fc32568606934360b531ca0d573b229b (127.21.140.65:39051)
I20260928 21:10:35.724403 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:43507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:10:35.737411 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:45233
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:35.850859 6925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:35.851148 6925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:35.851193 6925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:35.855356 6925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:35.855422 6925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:35.855484 6925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:35.860882 6925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:45233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:35.861910 6925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:35.863145 6925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:35.873252 6930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:35.873399 6925 server_base.cc:1034] running on GCE node
W20260928 21:10:35.873432 6931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:35.873237 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:35.874006 6925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:35.874672 6925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:35.875833 6925 hybrid_clock.cc:648] HybridClock initialized: now 1790629835875817 us; error 30 us; skew 500 ppm
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:35.878914 6925 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:10:35.879798 6925 webserver.cc:460] Webserver started at http://127.21.140.66:45597/ using document root <none> and password file <none>
I20260928 21:10:35.880357 6925 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:35.880417 6925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:35.880707 6925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:35.882956 6925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "45f45d82eb654d0f8c2a9bbec1db9b00"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "fd7cd6854d87e51461cc3fc1e8053ec4"
server_key_iv: "5128d129b236dcc8dc56a6a355c7c124"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.883550 6925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "45f45d82eb654d0f8c2a9bbec1db9b00"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "fd7cd6854d87e51461cc3fc1e8053ec4"
server_key_iv: "5128d129b236dcc8dc56a6a355c7c124"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.887394 6925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:10:35.890225 6940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.891528 6925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:35.891685 6925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "45f45d82eb654d0f8c2a9bbec1db9b00"
format_stamp: "Formatted at 2026-09-28 21:10:35 on dist-test-slave-d5wc"
server_key: "fd7cd6854d87e51461cc3fc1e8053ec4"
server_key_iv: "5128d129b236dcc8dc56a6a355c7c124"
server_key_version: "encryptionkey@0"
I20260928 21:10:35.891801 6925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:35.907246 6925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:35.910737 6925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:35.910959 6925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:35.911567 6925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:35.912415 6925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:35.912460 6925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.912523 6925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:35.912560 6925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:35.921949 6925 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:45773
I20260928 21:10:35.922055 7054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:45773 every 8 connection(s)
I20260928 21:10:35.922940 6925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:35.924490 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 6925
I20260928 21:10:35.924623 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:10:35.925010 22065 external_mini_cluster.cc:1428] Setting key d756fcaf67adcf3e4be615ebc22f14ee
Sep 28 21:10:35 dist-test-slave-d5wc krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629835, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:10:35.940171 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:35.924968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:39893 (local address 127.21.140.126:45233)
0928 21:10:35.925126 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:35.925130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:35.926364 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 21:10:35.927833 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:35.927836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:35.927889 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:35.927979 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:35.929211 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.930134 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.931467 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.931937 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.932068 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:35.934706 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:35.934711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:35.934716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:35.934748 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:35.936406 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:35.937178 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:35.937181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:35.937183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:35.937261 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:35.937740 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:35.937743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:35.937745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:35.938014 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:35.938137 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:35.939855 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0928 21:10:35.939986 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:10:35.940994 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:35.925502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:45233 (local address 127.21.140.66:39893)
0928 21:10:35.926183 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:35.926245 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:35.927614 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:35.928157 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:35.928164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:35.928542 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:35.929052 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:35.929066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.930278 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.930281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:35.931317 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:35.931330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:35.932103 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:35.932106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:35.932271 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:35.932918 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:35.932935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:35.934515 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:35.936592 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:35.936598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:35.936610 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:35.937048 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:35.937375 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:35.937378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:35.937379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:35.937610 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:35.939344 (+ 1734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:35.939350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:35.939606 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 21:10:35.939812 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 21:10:35.940144 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":142,"threads_started":1}
I20260928 21:10:35.942237 7055 heartbeater.cc:347] Connected to a master server at 127.21.140.126:45233
I20260928 21:10:35.942493 7055 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:35.942982 7055 heartbeater.cc:511] Master 127.21.140.126:45233 requested a full tablet report, sending...
I20260928 21:10:35.944087 6728 ts_manager.cc:194] Registered new tserver with Master: 45f45d82eb654d0f8c2a9bbec1db9b00 (127.21.140.66:45773)
I20260928 21:10:35.944932 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:39893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:10:35.954136 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:45233
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:36.070061 7060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:36.070305 7060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:36.070350 7060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:36.074326 7060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:36.074378 7060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:36.074435 7060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:36.079638 7060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:45233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:36.080825 7060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:36.081959 7060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:36.091869 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.091909 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.091869 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:36.092170 7060 server_base.cc:1034] running on GCE node
I20260928 21:10:36.092585 7060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:36.093220 7060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:36.094412 7060 hybrid_clock.cc:648] HybridClock initialized: now 1790629836094398 us; error 25 us; skew 500 ppm
Sep 28 21:10:36 dist-test-slave-d5wc krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629836, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:36.097374 7060 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:10:36.098287 7060 webserver.cc:460] Webserver started at http://127.21.140.67:44201/ using document root <none> and password file <none>
I20260928 21:10:36.098860 7060 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:36.098953 7060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:36.099193 7060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:36.101313 7060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "d6e051af2f4644d8ad6850e4596ac15d"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "12167eec294532a66ac2ece7aa735535"
server_key_iv: "1154f1e449a222a1093bbec63ce0ad31"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.101910 7060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "d6e051af2f4644d8ad6850e4596ac15d"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "12167eec294532a66ac2ece7aa735535"
server_key_iv: "1154f1e449a222a1093bbec63ce0ad31"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.105633 7060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:10:36.108223 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.109454 7060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:10:36.109591 7060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "d6e051af2f4644d8ad6850e4596ac15d"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "12167eec294532a66ac2ece7aa735535"
server_key_iv: "1154f1e449a222a1093bbec63ce0ad31"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.109720 7060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:36.127821 7060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:36.131366 7060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:36.131577 7060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:36.132180 7060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:36.133006 7060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:36.133093 7060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.133164 7060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:36.133213 7060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.142916 7060 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:36153
I20260928 21:10:36.142939 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:36153 every 8 connection(s)
I20260928 21:10:36.143839 7060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:10:36.151197 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7060
I20260928 21:10:36.151360 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:10:36.151715 22065 external_mini_cluster.cc:1428] Setting key 383c54c6036f188c40e8c6cd80597f1f
Sep 28 21:10:36 dist-test-slave-d5wc krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629836, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:36.158008 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.145617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:54389 (local address 127.21.140.126:45233)
0928 21:10:36.145759 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.145763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.146704 (+ 941us) server_negotiation.cc:407] Connection header received
0928 21:10:36.148100 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.148103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.148165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.148270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:36.149430 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.150021 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.151172 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.151482 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.151609 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.154553 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:36.154557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:36.154560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:36.154587 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:36.155950 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:36.156459 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:36.156464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:36.156466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:36.156559 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:36.156906 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:36.156909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:36.156911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:36.157105 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:36.157213 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.157705 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.157825 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:10:36.158841 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.145990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:45233 (local address 127.21.140.67:54389)
0928 21:10:36.146450 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.146493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.147870 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.148442 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.148449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.148793 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:36.149268 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.149282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.150162 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.150165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.151044 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.151052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.151652 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.151656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.151871 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.152563 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:36.152587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:36.154385 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:36.156076 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:36.156082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:36.156097 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:36.156353 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:36.156652 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:36.156657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:36.156659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:36.156803 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:36.157213 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:36.157221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:36.157470 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.157665 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.158019 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260928 21:10:36.159927 7190 heartbeater.cc:347] Connected to a master server at 127.21.140.126:45233
I20260928 21:10:36.160212 7190 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:36.160755 7190 heartbeater.cc:511] Master 127.21.140.126:45233 requested a full tablet report, sending...
I20260928 21:10:36.161753 6728 ts_manager.cc:194] Registered new tserver with Master: d6e051af2f4644d8ad6850e4596ac15d (127.21.140.67:36153)
I20260928 21:10:36.162382 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:54389
I20260928 21:10:36.165721 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:10:36.175057 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.167906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.21.140.126:45233)
0928 21:10:36.168042 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.168046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.168216 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:10:36.168315 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.168317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.168355 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.168426 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:36.169124 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.169625 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.170471 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.170718 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.170797 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.172188 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:36.172191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:36.172193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:36.172213 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:36.173417 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:36.173875 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:36.173877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:36.173879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:36.173950 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:36.174235 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:36.174237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:36.174238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:36.174453 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:36.174567 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.174766 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.174858 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:10:36.178069 6728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:10:36.181136 6728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:10:36.196194 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.190343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.21.140.65:39051)
0928 21:10:36.191271 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.191294 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.191317 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:10:36.191392 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.191397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.191593 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.191750 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:36.192127 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.192949 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.194693 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.195475 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.195599 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.195754 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.195922 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.196037 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":60,"threads_started":1}
I20260928 21:10:36.196534 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.189920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:39051 (local address 127.0.0.1:57924)
0928 21:10:36.190719 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.190733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.190829 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.191733 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.191736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.191764 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:36.191984 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.191991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.193101 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.193106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.194558 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.194568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.195600 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.195604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.195708 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.195753 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.195891 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.196114 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":100,"spinlock_wait_cycles":1920,"thread_start_us":152,"threads_started":1}
I20260928 21:10:36.196720 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.190586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:36153 (local address 127.0.0.1:33910)
0928 21:10:36.191382 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.191396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.191477 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.191980 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.191983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.192000 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:36.192092 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.192097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.193481 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.193485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.194739 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.194746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.196016 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.196019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.196205 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.196265 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.196340 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.196525 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":86,"thread_start_us":86,"threads_started":1}
I20260928 21:10:36.196844 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.190004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.21.140.66:45773)
0928 21:10:36.190583 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.190596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.190971 (+ 375us) server_negotiation.cc:407] Connection header received
0928 21:10:36.191057 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.191065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.191286 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.191444 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:36.192124 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.193336 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.194844 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.195706 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.195814 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.195938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.196606 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.196700 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":126,"threads_started":1}
I20260928 21:10:36.196844 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.190797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.21.140.67:36153)
0928 21:10:36.191126 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.191138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.191573 (+ 435us) server_negotiation.cc:407] Connection header received
0928 21:10:36.191654 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.191659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.191827 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.191979 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:36.192201 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.193336 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.194882 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.195706 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.195814 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.195938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.196606 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.196703 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":143,"threads_started":1}
I20260928 21:10:36.197111 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.189847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45773 (local address 127.0.0.1:43636)
0928 21:10:36.190626 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.190680 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.190829 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.191718 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.191724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.191748 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:36.191985 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.191994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.193529 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.193531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.194713 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.194723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.196007 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.196011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.196200 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.196266 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.196468 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.196538 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":204,"thread_start_us":81,"threads_started":1}
I20260928 21:10:36.199654 7124 tablet_service.cc:1467] Processing CreateTablet for tablet cb0e53039c0b427ea21d647ee10223c0 (DEFAULT_TABLE table=test-table [id=1acddd3d6e584cc7bc01b53b4bcc0734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:36.199781 6854 tablet_service.cc:1467] Processing CreateTablet for tablet cb0e53039c0b427ea21d647ee10223c0 (DEFAULT_TABLE table=test-table [id=1acddd3d6e584cc7bc01b53b4bcc0734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:36.199781 6988 tablet_service.cc:1467] Processing CreateTablet for tablet cb0e53039c0b427ea21d647ee10223c0 (DEFAULT_TABLE table=test-table [id=1acddd3d6e584cc7bc01b53b4bcc0734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:36.200697 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0e53039c0b427ea21d647ee10223c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:36.200795 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0e53039c0b427ea21d647ee10223c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:36.200795 6854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0e53039c0b427ea21d647ee10223c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:36.207748 7207 tablet_bootstrap.cc:492] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Bootstrap starting.
I20260928 21:10:36.208524 7208 tablet_bootstrap.cc:492] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b: Bootstrap starting.
I20260928 21:10:36.209831 7207 tablet_bootstrap.cc:654] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:36.210299 7208 tablet_bootstrap.cc:654] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:36.210310 7209 tablet_bootstrap.cc:492] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: Bootstrap starting.
I20260928 21:10:36.211261 7207 log.cc:824] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:36.211309 7208 log.cc:824] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:36.212435 7209 tablet_bootstrap.cc:654] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:36.213303 7208 tablet_bootstrap.cc:492] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b: No bootstrap required, opened a new log
I20260928 21:10:36.213502 7207 tablet_bootstrap.cc:492] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: No bootstrap required, opened a new log
I20260928 21:10:36.213526 7208 ts_tablet_manager.cc:1397] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:10:36.213696 7207 ts_tablet_manager.cc:1397] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 21:10:36.213877 7209 log.cc:824] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:36.215840 7209 tablet_bootstrap.cc:492] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: No bootstrap required, opened a new log
I20260928 21:10:36.216029 7209 ts_tablet_manager.cc:1397] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260928 21:10:36.216881 7207 raft_consensus.cc:348] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.217090 7207 raft_consensus.cc:374] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:36.217123 7207 raft_consensus.cc:729] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e051af2f4644d8ad6850e4596ac15d, State: Initialized, Role: FOLLOWER
I20260928 21:10:36.217424 7208 raft_consensus.cc:348] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.217557 7207 consensus_queue.cc:260] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.217693 7208 raft_consensus.cc:374] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:36.217749 7208 raft_consensus.cc:729] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc32568606934360b531ca0d573b229b, State: Initialized, Role: FOLLOWER
I20260928 21:10:36.218248 7190 heartbeater.cc:503] Master 127.21.140.126:45233 was elected leader, sending a full tablet report...
I20260928 21:10:36.218214 7208 consensus_queue.cc:260] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.218583 7207 ts_tablet_manager.cc:1428] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 21:10:36.219121 7208 ts_tablet_manager.cc:1428] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:10:36.219444 6920 heartbeater.cc:503] Master 127.21.140.126:45233 was elected leader, sending a full tablet report...
I20260928 21:10:36.220100 7209 raft_consensus.cc:348] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.220337 7209 raft_consensus.cc:374] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:36.220387 7209 raft_consensus.cc:729] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f45d82eb654d0f8c2a9bbec1db9b00, State: Initialized, Role: FOLLOWER
I20260928 21:10:36.220772 7209 consensus_queue.cc:260] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.221314 7055 heartbeater.cc:503] Master 127.21.140.126:45233 was elected leader, sending a full tablet report...
I20260928 21:10:36.221625 7209 ts_tablet_manager.cc:1428] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260928 21:10:36.226305 7214 raft_consensus.cc:482] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:36.226444 7214 raft_consensus.cc:504] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.227507 7214 leader_election.cc:288] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f45d82eb654d0f8c2a9bbec1db9b00 (127.21.140.66:45773), d6e051af2f4644d8ad6850e4596ac15d (127.21.140.67:36153)
I20260928 21:10:36.231258 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.227807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:49841 (local address 127.21.140.66:45773)
0928 21:10:36.227930 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.227934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.228111 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:10:36.228280 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.228283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.228325 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.228415 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:36.228695 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.229203 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.230311 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.230722 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.230850 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.230899 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.231082 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.231143 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:10:36.231276 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.227753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45773 (local address 127.21.140.65:49841)
0928 21:10:36.228027 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.228043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.228186 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.228429 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.228432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.228451 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:36.228583 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.228589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.229316 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.229320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.230192 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.230202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.230843 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.230846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.230941 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.230963 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.231055 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.231129 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260928 21:10:36.231674 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.227753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:36153 (local address 127.21.140.65:41607)
0928 21:10:36.228449 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.228461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.228581 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.228822 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.228825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.228840 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:36.228974 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.228980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.229702 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.229707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.230564 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.230574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.231198 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.231202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.231307 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.231336 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.231457 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.231528 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":342,"thread_start_us":104,"threads_started":1}
I20260928 21:10:36.231710 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.228045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:41607 (local address 127.21.140.67:36153)
0928 21:10:36.228161 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.228165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.228544 (+ 379us) server_negotiation.cc:407] Connection header received
0928 21:10:36.228692 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.228694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.228732 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.228816 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:36.229082 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.229590 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.230687 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.231078 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.231205 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.231251 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.231511 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.231590 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:10:36.231969 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0e53039c0b427ea21d647ee10223c0" candidate_uuid: "fc32568606934360b531ca0d573b229b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" is_pre_election: true
I20260928 21:10:36.232273 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0e53039c0b427ea21d647ee10223c0" candidate_uuid: "fc32568606934360b531ca0d573b229b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e051af2f4644d8ad6850e4596ac15d" is_pre_election: true
I20260928 21:10:36.232298 7008 raft_consensus.cc:2454] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc32568606934360b531ca0d573b229b in term 0.
I20260928 21:10:36.232564 7144 raft_consensus.cc:2454] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc32568606934360b531ca0d573b229b in term 0.
I20260928 21:10:36.232808 6810 leader_election.cc:302] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f45d82eb654d0f8c2a9bbec1db9b00, fc32568606934360b531ca0d573b229b; no voters:
I20260928 21:10:36.233108 7214 raft_consensus.cc:2781] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:10:36.233191 7214 raft_consensus.cc:482] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:36.233232 7214 raft_consensus.cc:3037] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:36.234362 7214 raft_consensus.cc:504] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.234707 7214 leader_election.cc:288] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [CANDIDATE]: Term 1 election: Requested vote from peers 45f45d82eb654d0f8c2a9bbec1db9b00 (127.21.140.66:45773), d6e051af2f4644d8ad6850e4596ac15d (127.21.140.67:36153)
I20260928 21:10:36.235075 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0e53039c0b427ea21d647ee10223c0" candidate_uuid: "fc32568606934360b531ca0d573b229b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00"
I20260928 21:10:36.235142 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0e53039c0b427ea21d647ee10223c0" candidate_uuid: "fc32568606934360b531ca0d573b229b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6e051af2f4644d8ad6850e4596ac15d"
I20260928 21:10:36.235220 7008 raft_consensus.cc:3037] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:36.235241 7144 raft_consensus.cc:3037] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:36.236413 7008 raft_consensus.cc:2454] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc32568606934360b531ca0d573b229b in term 1.
I20260928 21:10:36.236413 7144 raft_consensus.cc:2454] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc32568606934360b531ca0d573b229b in term 1.
I20260928 21:10:36.236781 6807 leader_election.cc:302] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6e051af2f4644d8ad6850e4596ac15d, fc32568606934360b531ca0d573b229b; no voters:
I20260928 21:10:36.236994 7214 raft_consensus.cc:2781] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:36.237214 7214 raft_consensus.cc:686] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [term 1 LEADER]: Becoming Leader. State: Replica: fc32568606934360b531ca0d573b229b, State: Running, Role: LEADER
I20260928 21:10:36.237623 7214 consensus_queue.cc:237] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } }
I20260928 21:10:36.240885 6727 catalog_manager.cc:5495] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b reported cstate change: term changed from 0 to 1, leader changed from <none> to fc32568606934360b531ca0d573b229b (127.21.140.65). New cstate: current_term: 1 leader_uuid: "fc32568606934360b531ca0d573b229b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc32568606934360b531ca0d573b229b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 39051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 } health_report { overall_health: UNKNOWN } } }
I20260928 21:10:36.259562 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.255999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.21.140.65:39051)
0928 21:10:36.256151 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.256155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.256172 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:10:36.256266 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.256269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.256323 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.256434 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:10:36.256686 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.257212 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.258111 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.258334 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.258448 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.258753 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:10:36.259157 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:10:36.259291 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.259353 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.259410 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:10:36.265511 7144 raft_consensus.cc:1261] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 1 FOLLOWER]: Refusing update from remote peer fc32568606934360b531ca0d573b229b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:36.265539 7008 raft_consensus.cc:1261] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 1 FOLLOWER]: Refusing update from remote peer fc32568606934360b531ca0d573b229b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:36.266371 7218 consensus_queue.cc:1035] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e051af2f4644d8ad6850e4596ac15d" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:36.266621 7214 consensus_queue.cc:1035] T cb0e53039c0b427ea21d647ee10223c0 P fc32568606934360b531ca0d573b229b [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f45d82eb654d0f8c2a9bbec1db9b00" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:36.277189 7223 mvcc.cc:205] Tried to move back new op lower bound from 7334419809335865344 to 7334419809232527360. Current Snapshot: MvccSnapshot[applied={T|T < 7334419809335865344}]
I20260928 21:10:36.279330 7221 mvcc.cc:205] Tried to move back new op lower bound from 7334419809335865344 to 7334419809232527360. Current Snapshot: MvccSnapshot[applied={T|T < 7334419809335865344 or (T in {7334419809335865344})}]
I20260928 21:10:36.284184 6727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:36.284370 6727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:36.286723 6727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b6010bffd92439ca6504448ec2b9528: Sending DeleteTablet for 3 replicas of tablet cb0e53039c0b427ea21d647ee10223c0
I20260928 21:10:36.287534 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0e53039c0b427ea21d647ee10223c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:33910
I20260928 21:10:36.287523 6988 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0e53039c0b427ea21d647ee10223c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:43636
I20260928 21:10:36.287997 7231 tablet_replica.cc:330] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: stopping tablet replica
I20260928 21:10:36.288307 7231 raft_consensus.cc:2229] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:36.288635 7231 raft_consensus.cc:2258] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:36.288581 6854 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0e53039c0b427ea21d647ee10223c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:36 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:57924
I20260928 21:10:36.288787 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 6791
I20260928 21:10:36.289533 7232 tablet_replica.cc:330] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: stopping tablet replica
I20260928 21:10:36.289844 7232 raft_consensus.cc:2229] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:36.290174 7232 raft_consensus.cc:2258] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:36.291617 7231 ts_tablet_manager.cc:1905] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:36.294090 7231 ts_tablet_manager.cc:1918] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:36.294214 7231 log.cc:1196] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/cb0e53039c0b427ea21d647ee10223c0
I20260928 21:10:36.294621 7231 ts_tablet_manager.cc:1939] T cb0e53039c0b427ea21d647ee10223c0 P d6e051af2f4644d8ad6850e4596ac15d: Deleting consensus metadata
I20260928 21:10:36.294632 7232 ts_tablet_manager.cc:1905] T cb0e53039c0b427ea21d647ee10223c0 P 45f45d82eb654d0f8c2a9bbec1db9b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:36.296198 6712 catalog_manager.cc:4841] TS d6e051af2f4644d8ad6850e4596ac15d (127.21.140.67:36153): tablet cb0e53039c0b427ea21d647ee10223c0 (table test-table [id=1acddd3d6e584cc7bc01b53b4bcc0734]) successfully deleted
W20260928 21:10:36.296976 6712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:39051 (error 108)
I20260928 21:10:36.297026 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 6925
I20260928 21:10:36.299211 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.298816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:39051 (local address 127.0.0.1:57936)
0928 21:10:36.298989 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.299089 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:39051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 21:10:36.299410 6712 catalog_manager.cc:4584] TS fc32568606934360b531ca0d573b229b (127.21.140.65:39051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb0e53039c0b427ea21d647ee10223c0: Network error: Client connection negotiation failed: client connection to 127.21.140.65:39051: connect: Connection refused (error 111)
I20260928 21:10:36.304936 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7060
I20260928 21:10:36.307006 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.305099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45773 (local address 127.0.0.1:43652)
0928 21:10:36.306807 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.306883 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1596}
I20260928 21:10:36.311406 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 6696
2026-09-28T21:10:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2318 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T21:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:36Z Disabled control of system clock
I20260928 21:10:36.342797 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37109
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:10:36.457693 7239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:36.458051 7239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:36.458107 7239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:36.462177 7239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:36.462239 7239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:36.462255 7239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:36.462272 7239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:36.462287 7239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:36.467133 7239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:36.468186 7239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:36.469617 7239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:36.478577 7244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.478730 7245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.478586 7247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:36.478600 7239 server_base.cc:1034] running on GCE node
I20260928 21:10:36.479583 7239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:36.480820 7239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:36.482062 7239 hybrid_clock.cc:648] HybridClock initialized: now 1790629836482010 us; error 67 us; skew 500 ppm
I20260928 21:10:36.484127 7239 webserver.cc:460] Webserver started at http://127.21.140.126:38483/ using document root <none> and password file <none>
I20260928 21:10:36.484737 7239 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:36.484833 7239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:36.485050 7239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:36.487212 7239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "a10034f3898c4fffb1b59339de6a7371"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "1ce94743822d5c7927344772b781f751"
server_key_iv: "2aa2268cddc10cf89c996b6ba5ba2548"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.487794 7239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "a10034f3898c4fffb1b59339de6a7371"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "1ce94743822d5c7927344772b781f751"
server_key_iv: "2aa2268cddc10cf89c996b6ba5ba2548"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.491604 7239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:36.494297 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.495656 7239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:36.495836 7239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "a10034f3898c4fffb1b59339de6a7371"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "1ce94743822d5c7927344772b781f751"
server_key_iv: "2aa2268cddc10cf89c996b6ba5ba2548"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.495975 7239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:36.506999 7239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:36.507711 7239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:36.507920 7239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:36.515059 7239 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37109
I20260928 21:10:36.515082 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37109 every 8 connection(s)
I20260928 21:10:36.516067 7239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:36.518682 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7239
I20260928 21:10:36.518813 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:36.519126 22065 external_mini_cluster.cc:1428] Setting key 36c36d69a80776530d1e6d589dabdd7b
I20260928 21:10:36.519639 7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:36.526173 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371: Bootstrap starting.
I20260928 21:10:36.528589 7306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:36.529284 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.520635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.21.140.126:37109)
0928 21:10:36.521292 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.521308 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.521345 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:10:36.522468 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.522488 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.522935 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.523288 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:36.523738 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.524717 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.525639 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.526173 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.526345 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.527352 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:36.527358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:36.527372 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:36.527794 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:36.527856 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:36.527877 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:36.528019 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.528146 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.528429 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":151,"threads_started":1}
I20260928 21:10:36.529330 7306 log.cc:824] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:36.531461 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371: No bootstrap required, opened a new log
I20260928 21:10:36.535207 7306 raft_consensus.cc:348] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10034f3898c4fffb1b59339de6a7371" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37109 } }
I20260928 21:10:36.535423 7306 raft_consensus.cc:374] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:36.535491 7306 raft_consensus.cc:729] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10034f3898c4fffb1b59339de6a7371, State: Initialized, Role: FOLLOWER
I20260928 21:10:36.535933 7306 consensus_queue.cc:260] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10034f3898c4fffb1b59339de6a7371" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37109 } }
I20260928 21:10:36.536085 7306 raft_consensus.cc:388] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:36.536172 7306 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:36.536286 7306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:36.537191 7306 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10034f3898c4fffb1b59339de6a7371" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37109 } }
I20260928 21:10:36.537526 7306 leader_election.cc:302] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a10034f3898c4fffb1b59339de6a7371; no voters:
I20260928 21:10:36.537840 7306 leader_election.cc:288] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:36.538012 7311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:36.538269 7311 raft_consensus.cc:686] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [term 1 LEADER]: Becoming Leader. State: Replica: a10034f3898c4fffb1b59339de6a7371, State: Running, Role: LEADER
I20260928 21:10:36.538590 7311 consensus_queue.cc:237] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10034f3898c4fffb1b59339de6a7371" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37109 } }
I20260928 21:10:36.539093 7306 sys_catalog.cc:564] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:36.540269 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a10034f3898c4fffb1b59339de6a7371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10034f3898c4fffb1b59339de6a7371" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37109 } } }
I20260928 21:10:36.540427 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:36.540325 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a10034f3898c4fffb1b59339de6a7371. Latest consensus state: current_term: 1 leader_uuid: "a10034f3898c4fffb1b59339de6a7371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10034f3898c4fffb1b59339de6a7371" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37109 } } }
I20260928 21:10:36.540714 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:36.540966 7320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:36.544350 7320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:36.550935 7320 catalog_manager.cc:1347] Generated new cluster ID: 1af8518739f942f190cdba17e95ebb78
I20260928 21:10:36.551044 7320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:36.560735 7320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:36.561630 7320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:36.568871 7320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371: Generated new TSK 0
I20260928 21:10:36.569607 7320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:36.576495 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37109
--builtin_ntp_servers=127.21.140.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:10:36.689810 7330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:36.690160 7330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:36.690243 7330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:36.694993 7330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:36.695077 7330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:36.695165 7330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:36.700294 7330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:36.701452 7330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:36.703023 7330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:36.712824 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.712863 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.712863 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:36.712924 7330 server_base.cc:1034] running on GCE node
I20260928 21:10:36.713600 7330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:36.714315 7330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:36.715512 7330 hybrid_clock.cc:648] HybridClock initialized: now 1790629836715491 us; error 32 us; skew 500 ppm
I20260928 21:10:36.717386 7330 webserver.cc:460] Webserver started at http://127.21.140.65:35511/ using document root <none> and password file <none>
I20260928 21:10:36.717917 7330 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:36.718047 7330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:36.718233 7330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:36.720197 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "ece166cb91e8407daaafaa3c84d2294d"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "c2a0cc5dcdcfe6f25eafd70fc3cc5024"
server_key_iv: "4a6eadde41653bbb58f240d38f2b6925"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.720734 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "ece166cb91e8407daaafaa3c84d2294d"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "c2a0cc5dcdcfe6f25eafd70fc3cc5024"
server_key_iv: "4a6eadde41653bbb58f240d38f2b6925"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.724462 7330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 21:10:36.727054 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.728384 7330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:36.728492 7330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "ece166cb91e8407daaafaa3c84d2294d"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "c2a0cc5dcdcfe6f25eafd70fc3cc5024"
server_key_iv: "4a6eadde41653bbb58f240d38f2b6925"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.728587 7330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:36.758901 7330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:36.759734 7330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:36.759960 7330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:36.760593 7330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:36.761565 7330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:36.761615 7330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.761682 7330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:36.761713 7330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.771698 7330 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:35147
I20260928 21:10:36.771718 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:35147 every 8 connection(s)
I20260928 21:10:36.772668 7330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:36.774289 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7330
I20260928 21:10:36.774405 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:10:36.774739 22065 external_mini_cluster.cc:1428] Setting key e88ae677e7e5ccd87485fd25e9e67a0e
I20260928 21:10:36.776831 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37109
--builtin_ntp_servers=127.21.140.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:10:36.783895 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.774835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:44203 (local address 127.21.140.126:37109)
0928 21:10:36.774976 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.774980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.776316 (+ 1336us) server_negotiation.cc:407] Connection header received
0928 21:10:36.777813 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.777819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.777887 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.778026 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:36.779292 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.780253 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.781410 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.781860 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.782023 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.783283 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:36.783291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:36.783297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:36.783317 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:36.783347 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:36.783360 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:36.783444 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.783605 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.783685 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:10:36.784642 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.775325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37109 (local address 127.21.140.65:44203)
0928 21:10:36.776134 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.776200 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.777609 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.778252 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.778264 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.778382 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:36.779089 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.779110 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.780393 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.780395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.781283 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.781292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.782085 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.782088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.782249 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.782904 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:36.782905 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:36.782943 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:36.783034 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:36.783462 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:36.783469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:36.783474 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.783570 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.783793 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":154,"threads_started":1}
I20260928 21:10:36.786022 7459 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37109
I20260928 21:10:36.786295 7459 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:36.786983 7459 heartbeater.cc:511] Master 127.21.140.126:37109 requested a full tablet report, sending...
I20260928 21:10:36.788584 7270 ts_manager.cc:194] Registered new tserver with Master: ece166cb91e8407daaafaa3c84d2294d (127.21.140.65:35147)
I20260928 21:10:36.789710 7270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.65:44203
W20260928 21:10:36.891291 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:36.891549 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:36.891588 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:36.895697 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:36.895756 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:36.895812 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:36.900796 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:36.901866 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:36.903285 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:36.913050 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.913023 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:36.913043 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:36.913393 7462 server_base.cc:1034] running on GCE node
I20260928 21:10:36.913905 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:36.914669 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:36.915870 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790629836915847 us; error 34 us; skew 500 ppm
I20260928 21:10:36.918133 7462 webserver.cc:460] Webserver started at http://127.21.140.66:38947/ using document root <none> and password file <none>
I20260928 21:10:36.918763 7462 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:36.918831 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:36.919060 7462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:36.921224 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "77a99b563a3f4d90ba5e00cbf72db141"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "bbad0ce6349fb9f52233296a35fde4b6"
server_key_iv: "dacafe981fbc6ad79239cff5be1f6228"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.921839 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "77a99b563a3f4d90ba5e00cbf72db141"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "bbad0ce6349fb9f52233296a35fde4b6"
server_key_iv: "dacafe981fbc6ad79239cff5be1f6228"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.925784 7462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 21:10:36.928402 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.929765 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:10:36.929924 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "77a99b563a3f4d90ba5e00cbf72db141"
format_stamp: "Formatted at 2026-09-28 21:10:36 on dist-test-slave-d5wc"
server_key: "bbad0ce6349fb9f52233296a35fde4b6"
server_key_iv: "dacafe981fbc6ad79239cff5be1f6228"
server_key_version: "encryptionkey@0"
I20260928 21:10:36.930117 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:36.950775 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:36.951520 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:36.951749 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:36.952358 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:36.953328 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:36.953404 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.953478 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:36.953522 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:36.962996 7462 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:45543
I20260928 21:10:36.963035 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:45543 every 8 connection(s)
I20260928 21:10:36.963941 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:36.973285 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.965807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:36601 (local address 127.21.140.126:37109)
0928 21:10:36.966008 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:36.966013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:36.967094 (+ 1081us) server_negotiation.cc:407] Connection header received
0928 21:10:36.968480 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:36.968482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:36.968532 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:36.968612 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:36.969568 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.970139 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.971358 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.971588 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.971679 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.972800 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:36.972803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:36.972805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:36.972818 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:36.972837 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:36.972845 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:36.972915 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:36.973112 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:10:36.973167 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:10:36.974304 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7462
I20260928 21:10:36.974433 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:10:36.974475 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:36.966255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37109 (local address 127.21.140.66:36601)
0928 21:10:36.966919 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:36.966963 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:36.968280 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:36.968775 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:36.968781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:36.968866 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:36.969424 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.969437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.970284 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.970287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.971217 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:36.971229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:36.971737 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:36.971740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:36.971884 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:36.972519 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:36.972521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:36.972563 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:36.972630 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:36.972962 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:36.972973 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:36.972978 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:10:36.973099 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 21:10:36.973429 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260928 21:10:36.974749 22065 external_mini_cluster.cc:1428] Setting key 918726cc1eb593df081903401fd7ce9c
I20260928 21:10:36.975963 7590 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37109
I20260928 21:10:36.976269 7590 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:36.976845 7590 heartbeater.cc:511] Master 127.21.140.126:37109 requested a full tablet report, sending...
I20260928 21:10:36.977136 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37109
--builtin_ntp_servers=127.21.140.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:10:36.978118 7270 ts_manager.cc:194] Registered new tserver with Master: 77a99b563a3f4d90ba5e00cbf72db141 (127.21.140.66:45543)
I20260928 21:10:36.978713 7270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.66:36601
W20260928 21:10:37.092768 7593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:37.093032 7593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:37.093071 7593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:37.097117 7593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:37.097182 7593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:37.097237 7593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:37.102366 7593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:37.103540 7593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:37.104892 7593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:37.114756 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:37.114756 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:37.114823 7593 server_base.cc:1034] running on GCE node
W20260928 21:10:37.114838 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:37.115429 7593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:37.116106 7593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:37.117321 7593 hybrid_clock.cc:648] HybridClock initialized: now 1790629837117284 us; error 49 us; skew 500 ppm
I20260928 21:10:37.119609 7593 webserver.cc:460] Webserver started at http://127.21.140.67:39839/ using document root <none> and password file <none>
I20260928 21:10:37.120280 7593 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:37.120388 7593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:37.120680 7593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:37.123018 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "c32cf3d9782c41608dc70a2f3a3b0de1"
format_stamp: "Formatted at 2026-09-28 21:10:37 on dist-test-slave-d5wc"
server_key: "4bcd55177975ee81eaabc390c5a56fcc"
server_key_iv: "b0fb0f191d00dbe1a5f088341feab3cc"
server_key_version: "encryptionkey@0"
I20260928 21:10:37.123662 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "c32cf3d9782c41608dc70a2f3a3b0de1"
format_stamp: "Formatted at 2026-09-28 21:10:37 on dist-test-slave-d5wc"
server_key: "4bcd55177975ee81eaabc390c5a56fcc"
server_key_iv: "b0fb0f191d00dbe1a5f088341feab3cc"
server_key_version: "encryptionkey@0"
I20260928 21:10:37.127789 7593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:37.130604 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:37.131893 7593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:37.132022 7593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "c32cf3d9782c41608dc70a2f3a3b0de1"
format_stamp: "Formatted at 2026-09-28 21:10:37 on dist-test-slave-d5wc"
server_key: "4bcd55177975ee81eaabc390c5a56fcc"
server_key_iv: "b0fb0f191d00dbe1a5f088341feab3cc"
server_key_version: "encryptionkey@0"
I20260928 21:10:37.132154 7593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:37.152511 7593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:37.153255 7593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:37.153476 7593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:37.154141 7593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:37.155102 7593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:37.155174 7593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:37.155256 7593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:37.155293 7593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:37.164321 7593 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:39911
I20260928 21:10:37.164395 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:39911 every 8 connection(s)
I20260928 21:10:37.165285 7593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:10:37.174301 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7593
I20260928 21:10:37.174434 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:10:37.174742 22065 external_mini_cluster.cc:1428] Setting key 61e77f3d535fc4abc081e9baef8f45e6
I20260928 21:10:37.175009 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.167333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:44413 (local address 127.21.140.126:37109)
0928 21:10:37.167494 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.167498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.168652 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 21:10:37.170131 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.170135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.170198 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.170309 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:37.171231 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.171775 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.172950 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.173184 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.173300 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.174476 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:37.174480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:37.174492 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:37.174511 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:37.174534 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:37.174552 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:37.174630 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.174800 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.174865 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:10:37.176074 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.167807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:37109 (local address 127.21.140.67:44413)
0928 21:10:37.168480 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.168529 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:37.169877 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:37.170495 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:37.170502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:37.170583 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:37.171065 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.171077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.171912 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.171916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.172810 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.172819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.173339 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.173343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.173513 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.174201 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:37.174203 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:37.174242 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:37.174308 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:37.174645 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:37.174656 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:37.174663 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:10:37.174795 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 21:10:37.175084 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260928 21:10:37.177218 7723 heartbeater.cc:347] Connected to a master server at 127.21.140.126:37109
I20260928 21:10:37.177426 7723 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:37.177889 7723 heartbeater.cc:511] Master 127.21.140.126:37109 requested a full tablet report, sending...
I20260928 21:10:37.178921 7270 ts_manager.cc:194] Registered new tserver with Master: c32cf3d9782c41608dc70a2f3a3b0de1 (127.21.140.67:39911)
I20260928 21:10:37.179574 7270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.67:44413
I20260928 21:10:37.188875 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:10:37.195264 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.190926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.21.140.126:37109)
0928 21:10:37.191147 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.191153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.191309 (+ 156us) server_negotiation.cc:407] Connection header received
0928 21:10:37.191478 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.191483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.191542 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.191641 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:37.191955 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.192557 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.193518 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.193765 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.193892 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.194737 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:37.194741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:37.194744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:37.194766 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:37.194785 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:37.194793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:37.194883 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.195034 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.195095 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 21:10:37.198472 7270 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:10:37.202147 7270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:10:37.217475 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.210663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45543 (local address 127.0.0.1:43066)
0928 21:10:37.211391 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.211441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:37.211620 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:37.212420 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:37.212426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:37.212464 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:37.212670 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.212678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.214172 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.214176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.215743 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.215749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.216921 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.216925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.217041 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.217083 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 21:10:37.217202 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:10:37.217313 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":119,"spinlock_wait_cycles":7168,"thread_start_us":107,"threads_started":1}
I20260928 21:10:37.217561 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.210853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.21.140.66:45543)
0928 21:10:37.211307 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.211324 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.211793 (+ 469us) server_negotiation.cc:407] Connection header received
0928 21:10:37.211880 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.211887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.212105 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.212254 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:37.212806 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.214040 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.215873 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.216763 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.216907 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.217110 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.217260 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.217367 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":120,"threads_started":1}
I20260928 21:10:37.218437 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.211395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.21.140.67:39911)
0928 21:10:37.211876 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.211886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.211904 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:10:37.211964 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.211968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.212104 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.212254 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:37.212806 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.214040 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.216932 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.217739 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.217842 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.218049 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.218165 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.218267 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":172,"threads_started":1}
I20260928 21:10:37.219161 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.211951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.21.140.65:35147)
0928 21:10:37.212428 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.212440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.212730 (+ 290us) server_negotiation.cc:407] Connection header received
0928 21:10:37.212876 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.212880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.213032 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.213194 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:37.213432 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.214426 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.217199 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.218298 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.218405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.218578 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.218922 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.219040 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":163,"threads_started":1}
I20260928 21:10:37.219162 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.211456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35147 (local address 127.0.0.1:39876)
0928 21:10:37.212615 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.212627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:37.212746 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:37.213183 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:37.213186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:37.213206 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:37.213302 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.213309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.214579 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.214583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.215739 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.215749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.218646 (+ 2897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.218650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.218748 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.218772 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:10:37.218854 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:10:37.219018 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":41,"thread_start_us":131,"threads_started":1}
I20260928 21:10:37.221349 7524 tablet_service.cc:1467] Processing CreateTablet for tablet d0d4f8d289ca4a7abfe18e0b990068f9 (DEFAULT_TABLE table=test-table [id=436fcb61d6e940a7af64d9a86ca99e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:37.222299 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.211020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:39911 (local address 127.0.0.1:57044)
0928 21:10:37.211616 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.211632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:37.211729 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:37.212420 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:37.212426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:37.212462 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:37.212677 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.212682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.215614 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.215617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.216798 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.216812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.217872 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.217876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.218013 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.218040 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:10:37.222044 (+ 4004us) client_negotiation.cc:240] Negotiation successful
0928 21:10:37.222137 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":63,"spinlock_wait_cycles":15872,"thread_start_us":125,"threads_started":1}
I20260928 21:10:37.221947 7393 tablet_service.cc:1467] Processing CreateTablet for tablet d0d4f8d289ca4a7abfe18e0b990068f9 (DEFAULT_TABLE table=test-table [id=436fcb61d6e940a7af64d9a86ca99e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:37.222664 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4f8d289ca4a7abfe18e0b990068f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:37.223052 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4f8d289ca4a7abfe18e0b990068f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:37.224617 7656 tablet_service.cc:1467] Processing CreateTablet for tablet d0d4f8d289ca4a7abfe18e0b990068f9 (DEFAULT_TABLE table=test-table [id=436fcb61d6e940a7af64d9a86ca99e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:37.225414 7656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4f8d289ca4a7abfe18e0b990068f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:37.230852 7740 tablet_bootstrap.cc:492] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: Bootstrap starting.
I20260928 21:10:37.232366 7741 tablet_bootstrap.cc:492] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Bootstrap starting.
I20260928 21:10:37.234112 7740 tablet_bootstrap.cc:654] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:37.234720 7741 tablet_bootstrap.cc:654] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:37.234962 7740 log.cc:824] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:37.235589 7741 log.cc:824] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:37.235651 7742 tablet_bootstrap.cc:492] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Bootstrap starting.
I20260928 21:10:37.237064 7740 tablet_bootstrap.cc:492] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: No bootstrap required, opened a new log
I20260928 21:10:37.237293 7740 ts_tablet_manager.cc:1397] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 21:10:37.237509 7741 tablet_bootstrap.cc:492] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: No bootstrap required, opened a new log
I20260928 21:10:37.237715 7741 ts_tablet_manager.cc:1397] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:10:37.241312 7740 raft_consensus.cc:348] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.241557 7740 raft_consensus.cc:374] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:37.241591 7740 raft_consensus.cc:729] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece166cb91e8407daaafaa3c84d2294d, State: Initialized, Role: FOLLOWER
I20260928 21:10:37.242141 7740 consensus_queue.cc:260] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.242388 7741 raft_consensus.cc:348] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.242693 7741 raft_consensus.cc:374] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:37.242761 7741 raft_consensus.cc:729] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a99b563a3f4d90ba5e00cbf72db141, State: Initialized, Role: FOLLOWER
I20260928 21:10:37.242995 7459 heartbeater.cc:503] Master 127.21.140.126:37109 was elected leader, sending a full tablet report...
I20260928 21:10:37.243148 7740 ts_tablet_manager.cc:1428] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 21:10:37.243284 7741 consensus_queue.cc:260] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.243986 7590 heartbeater.cc:503] Master 127.21.140.126:37109 was elected leader, sending a full tablet report...
I20260928 21:10:37.244377 7741 ts_tablet_manager.cc:1428] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:10:37.246840 7742 tablet_bootstrap.cc:654] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:37.247869 7742 log.cc:824] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:37.250039 7742 tablet_bootstrap.cc:492] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: No bootstrap required, opened a new log
I20260928 21:10:37.250260 7742 ts_tablet_manager.cc:1397] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260928 21:10:37.253508 7742 raft_consensus.cc:348] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.253743 7742 raft_consensus.cc:374] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:37.253777 7742 raft_consensus.cc:729] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32cf3d9782c41608dc70a2f3a3b0de1, State: Initialized, Role: FOLLOWER
I20260928 21:10:37.254320 7742 consensus_queue.cc:260] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.255033 7723 heartbeater.cc:503] Master 127.21.140.126:37109 was elected leader, sending a full tablet report...
I20260928 21:10:37.255324 7742 ts_tablet_manager.cc:1428] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:10:37.273586 7748 raft_consensus.cc:482] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:37.273810 7748 raft_consensus.cc:504] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
W20260928 21:10:37.273897 7460 tablet.cc:2369] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:10:37.275009 7748 leader_election.cc:288] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77a99b563a3f4d90ba5e00cbf72db141 (127.21.140.66:45543), ece166cb91e8407daaafaa3c84d2294d (127.21.140.65:35147)
I20260928 21:10:37.279474 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.275496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:43301 (local address 127.21.140.66:45543)
0928 21:10:37.275646 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.275651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.275668 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:10:37.275739 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.275743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.275800 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.275904 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:37.276254 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.276921 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.278332 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.278847 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.278944 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.278982 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.279238 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.279336 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:10:37.279474 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.275302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45543 (local address 127.21.140.67:43301)
0928 21:10:37.275421 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.275438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:37.275529 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:37.275940 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:37.275944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:37.275964 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:37.276111 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.276121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.277039 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.277043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.278197 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.278211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.278972 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.278976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.279092 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.279121 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 21:10:37.279216 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:10:37.279315 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 21:10:37.279951 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.275569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35147 (local address 127.21.140.67:59951)
0928 21:10:37.275928 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.275944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:37.276034 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:37.276409 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:37.276412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:37.276430 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:37.276543 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.276548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.277555 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.277557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.278521 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:37.278532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.279346 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.279349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:37.279445 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.279466 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:10:37.279713 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 21:10:37.279785 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":156,"threads_started":1}
I20260928 21:10:37.279958 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.276026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:59951 (local address 127.21.140.65:35147)
0928 21:10:37.276175 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.276180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.276198 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:10:37.276251 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.276254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.276309 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.276371 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:37.276698 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.277419 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.278662 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.279214 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.279332 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.279376 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.279740 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.279829 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:10:37.280311 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4f8d289ca4a7abfe18e0b990068f9" candidate_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a99b563a3f4d90ba5e00cbf72db141" is_pre_election: true
I20260928 21:10:37.280580 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4f8d289ca4a7abfe18e0b990068f9" candidate_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece166cb91e8407daaafaa3c84d2294d" is_pre_election: true
I20260928 21:10:37.280676 7544 raft_consensus.cc:2454] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c32cf3d9782c41608dc70a2f3a3b0de1 in term 0.
I20260928 21:10:37.280854 7413 raft_consensus.cc:2454] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c32cf3d9782c41608dc70a2f3a3b0de1 in term 0.
I20260928 21:10:37.281213 7609 leader_election.cc:302] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a99b563a3f4d90ba5e00cbf72db141, c32cf3d9782c41608dc70a2f3a3b0de1; no voters:
I20260928 21:10:37.281494 7748 raft_consensus.cc:2781] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:10:37.281574 7748 raft_consensus.cc:482] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:37.281612 7748 raft_consensus.cc:3037] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:37.282694 7748 raft_consensus.cc:504] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.283043 7748 leader_election.cc:288] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [CANDIDATE]: Term 1 election: Requested vote from peers 77a99b563a3f4d90ba5e00cbf72db141 (127.21.140.66:45543), ece166cb91e8407daaafaa3c84d2294d (127.21.140.65:35147)
I20260928 21:10:37.283506 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4f8d289ca4a7abfe18e0b990068f9" candidate_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a99b563a3f4d90ba5e00cbf72db141"
I20260928 21:10:37.283556 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4f8d289ca4a7abfe18e0b990068f9" candidate_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece166cb91e8407daaafaa3c84d2294d"
I20260928 21:10:37.283660 7544 raft_consensus.cc:3037] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:37.283752 7413 raft_consensus.cc:3037] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:37.284909 7413 raft_consensus.cc:2454] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c32cf3d9782c41608dc70a2f3a3b0de1 in term 1.
I20260928 21:10:37.285166 7544 raft_consensus.cc:2454] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c32cf3d9782c41608dc70a2f3a3b0de1 in term 1.
I20260928 21:10:37.285334 7609 leader_election.cc:302] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c32cf3d9782c41608dc70a2f3a3b0de1, ece166cb91e8407daaafaa3c84d2294d; no voters:
I20260928 21:10:37.285567 7748 raft_consensus.cc:2781] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:37.285813 7748 raft_consensus.cc:686] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 1 LEADER]: Becoming Leader. State: Replica: c32cf3d9782c41608dc70a2f3a3b0de1, State: Running, Role: LEADER
I20260928 21:10:37.286186 7748 consensus_queue.cc:237] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } }
I20260928 21:10:37.289517 7268 catalog_manager.cc:5495] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c32cf3d9782c41608dc70a2f3a3b0de1 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c32cf3d9782c41608dc70a2f3a3b0de1" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 39911 } health_report { overall_health: HEALTHY } } }
I20260928 21:10:37.316965 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.313334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.21.140.67:39911)
0928 21:10:37.313468 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:37.313472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:37.313488 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:10:37.313605 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:37.313607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:37.313663 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:37.313776 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:10:37.314043 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.314573 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.315489 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:37.315711 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:37.315826 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:37.316134 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:10:37.316536 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:10:37.316683 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:37.316743 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:10:37.316797 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:10:37.322973 7413 raft_consensus.cc:1261] T d0d4f8d289ca4a7abfe18e0b990068f9 P ece166cb91e8407daaafaa3c84d2294d [term 1 FOLLOWER]: Refusing update from remote peer c32cf3d9782c41608dc70a2f3a3b0de1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:37.322973 7544 raft_consensus.cc:1261] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 1 FOLLOWER]: Refusing update from remote peer c32cf3d9782c41608dc70a2f3a3b0de1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:37.323753 7750 consensus_queue.cc:1035] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece166cb91e8407daaafaa3c84d2294d" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:37.323946 7748 consensus_queue.cc:1035] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a99b563a3f4d90ba5e00cbf72db141" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:37.333676 7754 mvcc.cc:205] Tried to move back new op lower bound from 7334419813666459648 to 7334419813527023616. Current Snapshot: MvccSnapshot[applied={T|T < 7334419813666459648}]
I20260928 21:10:37.334146 7755 mvcc.cc:205] Tried to move back new op lower bound from 7334419813666459648 to 7334419813527023616. Current Snapshot: MvccSnapshot[applied={T|T < 7334419813666459648}]
I20260928 21:10:37.340886 7268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:37.341075 7268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:37.343686 7268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a10034f3898c4fffb1b59339de6a7371: Sending DeleteTablet for 3 replicas of tablet d0d4f8d289ca4a7abfe18e0b990068f9
I20260928 21:10:37.344630 7656 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d4f8d289ca4a7abfe18e0b990068f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:37 UTC) from {username='slave'} at 127.0.0.1:57044
I20260928 21:10:37.344784 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d4f8d289ca4a7abfe18e0b990068f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:37 UTC) from {username='slave'} at 127.0.0.1:43066
I20260928 21:10:37.345173 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7330
I20260928 21:10:37.345278 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d4f8d289ca4a7abfe18e0b990068f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:37 UTC) from {username='slave'} at 127.0.0.1:39876
I20260928 21:10:37.345903 7763 tablet_replica.cc:330] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: stopping tablet replica
I20260928 21:10:37.346297 7763 raft_consensus.cc:2229] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:37.346666 7763 raft_consensus.cc:2258] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:37.347049 7762 tablet_replica.cc:330] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: stopping tablet replica
I20260928 21:10:37.347359 7762 raft_consensus.cc:2229] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:10:37.347784 7762 raft_consensus.cc:2258] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:37.349328 7763 ts_tablet_manager.cc:1905] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:37.353101 7763 ts_tablet_manager.cc:1918] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:37.353230 7763 log.cc:1196] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/d0d4f8d289ca4a7abfe18e0b990068f9
I20260928 21:10:37.353556 7763 ts_tablet_manager.cc:1939] T d0d4f8d289ca4a7abfe18e0b990068f9 P 77a99b563a3f4d90ba5e00cbf72db141: Deleting consensus metadata
I20260928 21:10:37.354557 7762 ts_tablet_manager.cc:1905] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:37.355005 7255 catalog_manager.cc:4841] TS 77a99b563a3f4d90ba5e00cbf72db141 (127.21.140.66:45543): tablet d0d4f8d289ca4a7abfe18e0b990068f9 (table test-table [id=436fcb61d6e940a7af64d9a86ca99e53]) successfully deleted
I20260928 21:10:37.355348 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7462
W20260928 21:10:37.355335 7255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:35147 (error 108)
I20260928 21:10:37.357465 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:37.357150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:35147 (local address 127.0.0.1:39880)
0928 21:10:37.357289 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:37.357370 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 21:10:37.357642 7255 catalog_manager.cc:4584] TS ece166cb91e8407daaafaa3c84d2294d (127.21.140.65:35147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0d4f8d289ca4a7abfe18e0b990068f9: Network error: Client connection negotiation failed: client connection to 127.21.140.65:35147: connect: Connection refused (error 111)
I20260928 21:10:37.359162 7762 ts_tablet_manager.cc:1918] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:37.359272 7762 log.cc:1196] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/d0d4f8d289ca4a7abfe18e0b990068f9
I20260928 21:10:37.359586 7762 ts_tablet_manager.cc:1939] T d0d4f8d289ca4a7abfe18e0b990068f9 P c32cf3d9782c41608dc70a2f3a3b0de1: Deleting consensus metadata
I20260928 21:10:37.361101 7256 catalog_manager.cc:4841] TS c32cf3d9782c41608dc70a2f3a3b0de1 (127.21.140.67:39911): tablet d0d4f8d289ca4a7abfe18e0b990068f9 (table test-table [id=436fcb61d6e940a7af64d9a86ca99e53]) successfully deleted
I20260928 21:10:37.362475 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7593
I20260928 21:10:37.368227 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7239
2026-09-28T21:10:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1056 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:37 dist-test-slave-d5wc krb5kdc[7767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:37 dist-test-slave-d5wc krb5kdc[7767](info): set up 2 sockets
Sep 28 21:10:37 dist-test-slave-d5wc krb5kdc[7767](info): commencing operation
krb5kdc: starting...
W20260928 21:10:38.415820 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:38 dist-test-slave-d5wc krb5kdc[7767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:10:38.545203 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33749
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:38.656404 7780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:38.656692 7780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:38.656733 7780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:38.660742 7780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:38.660799 7780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:38.660818 7780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:38.660835 7780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:38.660851 7780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:38.666114 7780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:33749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:38.667133 7780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:38.668088 7780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:38.676970 7788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:38.676970 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:38.677001 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:38.677349 7780 server_base.cc:1034] running on GCE node
I20260928 21:10:38.678056 7780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:38.679256 7780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:38.680446 7780 hybrid_clock.cc:648] HybridClock initialized: now 1790629838680408 us; error 45 us; skew 500 ppm
Sep 28 21:10:38 dist-test-slave-d5wc krb5kdc[7767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629838, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:38.683300 7780 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:10:38.684312 7780 webserver.cc:460] Webserver started at http://127.21.140.126:40945/ using document root <none> and password file <none>
I20260928 21:10:38.684892 7780 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:38.684994 7780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:38.685237 7780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:38.687336 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "d96fce78026d4067ac7705984d58f8ab"
format_stamp: "Formatted at 2026-09-28 21:10:38 on dist-test-slave-d5wc"
server_key: "d11039c0062f24bc1ccdd8b5aca347d2"
server_key_iv: "1ff0122f2a726bea4aa5bfb78e0f9c40"
server_key_version: "encryptionkey@0"
I20260928 21:10:38.687996 7780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "d96fce78026d4067ac7705984d58f8ab"
format_stamp: "Formatted at 2026-09-28 21:10:38 on dist-test-slave-d5wc"
server_key: "d11039c0062f24bc1ccdd8b5aca347d2"
server_key_iv: "1ff0122f2a726bea4aa5bfb78e0f9c40"
server_key_version: "encryptionkey@0"
I20260928 21:10:38.692070 7780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:38.694561 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:38.695828 7780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:38.696007 7780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "d96fce78026d4067ac7705984d58f8ab"
format_stamp: "Formatted at 2026-09-28 21:10:38 on dist-test-slave-d5wc"
server_key: "d11039c0062f24bc1ccdd8b5aca347d2"
server_key_iv: "1ff0122f2a726bea4aa5bfb78e0f9c40"
server_key_version: "encryptionkey@0"
I20260928 21:10:38.696141 7780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:38.709822 7780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:38.712888 7780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:38.713112 7780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:38.720326 7780 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:33749
I20260928 21:10:38.720402 7848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:33749 every 8 connection(s)
I20260928 21:10:38.721355 7780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:38.721768 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7780
I20260928 21:10:38.721927 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:38.722254 22065 external_mini_cluster.cc:1428] Setting key fb3a13ea2c050e9636e7f29f86896df8
I20260928 21:10:38.724525 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:38.730106 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab: Bootstrap starting.
Sep 28 21:10:38 dist-test-slave-d5wc krb5kdc[7767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:38.732434 7849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:38.733193 7849 log.cc:824] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:38.735481 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab: No bootstrap required, opened a new log
I20260928 21:10:38.736783 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:38.723682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.21.140.126:33749)
0928 21:10:38.724141 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:38.724156 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:38.724191 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:10:38.724908 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:38.724926 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:38.725263 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:38.725603 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:38.726453 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:38.727472 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:38.728353 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:38.728826 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:38.729000 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:38.731686 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:38.731691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:38.731705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:38.731735 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:38.733537 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:38.734142 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:38.734147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:38.734152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:38.734271 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:38.734621 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:38.734625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:38.734626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:38.735086 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:38.735254 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:38.735696 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 21:10:38.736055 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260928 21:10:38.739230 7849 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96fce78026d4067ac7705984d58f8ab" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33749 } }
I20260928 21:10:38.739460 7849 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:38.739522 7849 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96fce78026d4067ac7705984d58f8ab, State: Initialized, Role: FOLLOWER
I20260928 21:10:38.739974 7849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96fce78026d4067ac7705984d58f8ab" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33749 } }
I20260928 21:10:38.740103 7849 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:38.740185 7849 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:38.740300 7849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:38.741291 7849 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96fce78026d4067ac7705984d58f8ab" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33749 } }
I20260928 21:10:38.741657 7849 leader_election.cc:302] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96fce78026d4067ac7705984d58f8ab; no voters:
I20260928 21:10:38.742034 7849 leader_election.cc:288] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:38.742177 7854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:38.742446 7854 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [term 1 LEADER]: Becoming Leader. State: Replica: d96fce78026d4067ac7705984d58f8ab, State: Running, Role: LEADER
I20260928 21:10:38.742811 7854 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96fce78026d4067ac7705984d58f8ab" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33749 } }
I20260928 21:10:38.743361 7849 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:38.744488 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96fce78026d4067ac7705984d58f8ab. Latest consensus state: current_term: 1 leader_uuid: "d96fce78026d4067ac7705984d58f8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96fce78026d4067ac7705984d58f8ab" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33749 } } }
I20260928 21:10:38.744638 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:38.745040 7863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:38.744985 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96fce78026d4067ac7705984d58f8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96fce78026d4067ac7705984d58f8ab" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33749 } } }
I20260928 21:10:38.745153 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:38.748476 7863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:38.754873 7863 catalog_manager.cc:1347] Generated new cluster ID: 55965a4b718549c7b13dd77348ffc5d2
I20260928 21:10:38.754967 7863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:38.783562 7863 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:38.784574 7863 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:38.789096 7863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab: Generated new TSK 0
I20260928 21:10:38.789793 7863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:10:38.830143 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33749
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:38.945845 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:38.946135 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:38.946182 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:38.950178 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:38.950239 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:38.950299 7875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:38.955708 7875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:38.956817 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:38.957778 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:38.967767 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:38.967801 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:38.968020 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:38.967885 7875 server_base.cc:1034] running on GCE node
I20260928 21:10:38.968480 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:38.969129 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:38.970290 7875 hybrid_clock.cc:648] HybridClock initialized: now 1790629838970276 us; error 27 us; skew 500 ppm
Sep 28 21:10:38 dist-test-slave-d5wc krb5kdc[7767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629838, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:38.973117 7875 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:10:38.973949 7875 webserver.cc:460] Webserver started at http://127.21.140.65:33189/ using document root <none> and password file <none>
I20260928 21:10:38.974550 7875 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:38.974639 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:38.974804 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:38.977301 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "9420f256ed4e4992b2b0e130a383a96e"
format_stamp: "Formatted at 2026-09-28 21:10:38 on dist-test-slave-d5wc"
server_key: "94c8e9c0b17068820c9f71c4fd9a227a"
server_key_iv: "5d3765b6cd74c453078b2d582971aca1"
server_key_version: "encryptionkey@0"
I20260928 21:10:38.977843 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "9420f256ed4e4992b2b0e130a383a96e"
format_stamp: "Formatted at 2026-09-28 21:10:38 on dist-test-slave-d5wc"
server_key: "94c8e9c0b17068820c9f71c4fd9a227a"
server_key_iv: "5d3765b6cd74c453078b2d582971aca1"
server_key_version: "encryptionkey@0"
I20260928 21:10:38.981611 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 21:10:38.984169 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:38.985507 7875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:10:38.985670 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "9420f256ed4e4992b2b0e130a383a96e"
format_stamp: "Formatted at 2026-09-28 21:10:38 on dist-test-slave-d5wc"
server_key: "94c8e9c0b17068820c9f71c4fd9a227a"
server_key_iv: "5d3765b6cd74c453078b2d582971aca1"
server_key_version: "encryptionkey@0"
I20260928 21:10:38.985797 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:39.022115 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:39.025617 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:39.025840 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:39.026499 7875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:39.027366 7875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:39.027449 7875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.027547 7875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:39.027596 7875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.036669 7875 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:33815
I20260928 21:10:39.036688 8003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:33815 every 8 connection(s)
I20260928 21:10:39.037607 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:39.047592 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 7875
Sep 28 21:10:39 dist-test-slave-d5wc krb5kdc[7767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629838, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:39.047751 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:10:39.048048 22065 external_mini_cluster.cc:1428] Setting key bee2c3ea9b5a42a826b55beed7b00850
I20260928 21:10:39.052553 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.039541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:58913 (local address 127.21.140.126:33749)
0928 21:10:39.039722 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.039727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.040660 (+ 933us) server_negotiation.cc:407] Connection header received
0928 21:10:39.042069 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.042073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.042121 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.042197 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:39.043453 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.044090 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.045402 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.045636 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.045723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.048381 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:39.048385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:39.048388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:39.048418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:39.050087 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.050741 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.050744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.050745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.050820 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.051260 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.051264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.051268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.051547 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:39.051672 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.052175 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.052337 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 21:10:39.053256 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.040013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33749 (local address 127.21.140.65:58913)
0928 21:10:39.040495 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.040539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.041785 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.042364 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.042371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.042749 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:39.043301 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.043322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.044210 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.044213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.045281 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.045291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.045775 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.045777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.045919 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.046565 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:39.046582 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:39.048199 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:39.050206 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:39.050217 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:39.050238 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:39.050642 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:39.050915 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:39.050920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:39.050925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:39.051135 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:39.051666 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:39.051671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:39.051914 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.052085 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.052350 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260928 21:10:39.054728 8004 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33749
I20260928 21:10:39.055016 8004 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:39.055721 8004 heartbeater.cc:511] Master 127.21.140.126:33749 requested a full tablet report, sending...
I20260928 21:10:39.057420 7812 ts_manager.cc:194] Registered new tserver with Master: 9420f256ed4e4992b2b0e130a383a96e (127.21.140.65:33815)
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:10:39.058700 7812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:58913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:10:39.076936 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33749
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:39.192869 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:39.193187 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:39.193274 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:39.197542 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:39.197651 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:39.197754 8009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:39.203563 8009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:39.204685 8009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:39.205735 8009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:39.215457 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:39.215487 8015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:39.215570 8009 server_base.cc:1034] running on GCE node
W20260928 21:10:39.215459 8014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:39.216182 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:39.216833 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:39.218053 8009 hybrid_clock.cc:648] HybridClock initialized: now 1790629839218013 us; error 53 us; skew 500 ppm
Sep 28 21:10:39 dist-test-slave-d5wc krb5kdc[7767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629839, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:39.220849 8009 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:10:39.221771 8009 webserver.cc:460] Webserver started at http://127.21.140.66:42337/ using document root <none> and password file <none>
I20260928 21:10:39.222411 8009 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:39.222510 8009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:39.222751 8009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:39.224856 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "ebee74c17fe04c4b8f7630175c457abb"
format_stamp: "Formatted at 2026-09-28 21:10:39 on dist-test-slave-d5wc"
server_key: "11126650e7489a922a6b944293177e4d"
server_key_iv: "00d32ee55a7ecfdeaf0fa72a0e250f3c"
server_key_version: "encryptionkey@0"
I20260928 21:10:39.225642 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "ebee74c17fe04c4b8f7630175c457abb"
format_stamp: "Formatted at 2026-09-28 21:10:39 on dist-test-slave-d5wc"
server_key: "11126650e7489a922a6b944293177e4d"
server_key_iv: "00d32ee55a7ecfdeaf0fa72a0e250f3c"
server_key_version: "encryptionkey@0"
I20260928 21:10:39.229447 8009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:39.232028 8024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.233489 8009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 21:10:39.233656 8009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "ebee74c17fe04c4b8f7630175c457abb"
format_stamp: "Formatted at 2026-09-28 21:10:39 on dist-test-slave-d5wc"
server_key: "11126650e7489a922a6b944293177e4d"
server_key_iv: "00d32ee55a7ecfdeaf0fa72a0e250f3c"
server_key_version: "encryptionkey@0"
I20260928 21:10:39.233785 8009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:39.256340 8009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:39.259574 8009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:39.259796 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:39.260396 8009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:39.261322 8009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:39.261407 8009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.261504 8009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:39.261557 8009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.270607 8009 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:39699
I20260928 21:10:39.270632 8139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:39699 every 8 connection(s)
I20260928 21:10:39.271551 8009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:39.273840 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8009
I20260928 21:10:39.274009 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:10:39.274309 22065 external_mini_cluster.cc:1428] Setting key 3b384c7acd62b0b80041be68b93d5467
Sep 28 21:10:39 dist-test-slave-d5wc krb5kdc[7767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629839, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:10:39.286998 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.273389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:35337 (local address 127.21.140.126:33749)
0928 21:10:39.273582 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.273586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.274614 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 21:10:39.276214 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.276217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.276272 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.276373 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:39.277565 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.278181 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.279357 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.279600 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.279716 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.282616 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:39.282623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:39.282627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:39.282658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:39.284310 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.284988 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.284991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.284992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.285162 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.285573 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.285576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.285577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.285768 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:39.285850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.286564 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.286730 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:10:39.287962 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.273867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33749 (local address 127.21.140.66:35337)
0928 21:10:39.274432 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.274489 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.275997 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.276534 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.276541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.276883 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:39.277401 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.277415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.278313 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.278316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.279220 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.279229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.279759 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.279763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.279956 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.280605 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:39.280622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:39.282407 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:39.284442 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:39.284448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:39.284470 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:39.284786 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:39.285269 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:39.285272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:39.285273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:39.285417 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:39.285905 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:39.285915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:39.286230 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.286431 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.286828 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":166,"threads_started":1}
I20260928 21:10:39.289371 8140 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33749
I20260928 21:10:39.289664 8140 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:39.290347 8140 heartbeater.cc:511] Master 127.21.140.126:33749 requested a full tablet report, sending...
I20260928 21:10:39.291617 7812 ts_manager.cc:194] Registered new tserver with Master: ebee74c17fe04c4b8f7630175c457abb (127.21.140.66:39699)
I20260928 21:10:39.292230 7812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:35337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:10:39.306120 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33749
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:39.423484 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:39.423779 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:39.423825 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:39.428073 8145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:39.428145 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:39.428231 8145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:39.433786 8145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:39.434959 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:39.436012 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:39.446637 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:39.446653 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:39.446650 8153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:39.446856 8145 server_base.cc:1034] running on GCE node
I20260928 21:10:39.447336 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:39.448024 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:39.449247 8145 hybrid_clock.cc:648] HybridClock initialized: now 1790629839449211 us; error 48 us; skew 500 ppm
Sep 28 21:10:39 dist-test-slave-d5wc krb5kdc[7767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629839, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:39.452353 8145 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:10:39.453302 8145 webserver.cc:460] Webserver started at http://127.21.140.67:39317/ using document root <none> and password file <none>
I20260928 21:10:39.453902 8145 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:39.454020 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:39.454282 8145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:39.456408 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "31a1137016d747cca0aa39c55c7d5fdc"
format_stamp: "Formatted at 2026-09-28 21:10:39 on dist-test-slave-d5wc"
server_key: "e8e7f7a4f2a9609d9145af76b7375686"
server_key_iv: "82dadad76523464a84479ec6ddb1d40c"
server_key_version: "encryptionkey@0"
I20260928 21:10:39.457006 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "31a1137016d747cca0aa39c55c7d5fdc"
format_stamp: "Formatted at 2026-09-28 21:10:39 on dist-test-slave-d5wc"
server_key: "e8e7f7a4f2a9609d9145af76b7375686"
server_key_iv: "82dadad76523464a84479ec6ddb1d40c"
server_key_version: "encryptionkey@0"
I20260928 21:10:39.460922 8145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:39.463608 8160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.464953 8145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:10:39.465124 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "31a1137016d747cca0aa39c55c7d5fdc"
format_stamp: "Formatted at 2026-09-28 21:10:39 on dist-test-slave-d5wc"
server_key: "e8e7f7a4f2a9609d9145af76b7375686"
server_key_iv: "82dadad76523464a84479ec6ddb1d40c"
server_key_version: "encryptionkey@0"
I20260928 21:10:39.465260 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:39.481426 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:39.484907 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:39.485126 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:39.485754 8145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:39.486663 8145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:39.486737 8145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.486810 8145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:39.486861 8145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:39.496558 8145 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:35433
I20260928 21:10:39.496589 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:35433 every 8 connection(s)
I20260928 21:10:39.497499 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:10:39.503261 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8145
I20260928 21:10:39.503429 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:10:39.503756 22065 external_mini_cluster.cc:1428] Setting key c2cddd8ed8834ab7bb6f855c9d1d7cac
Sep 28 21:10:39 dist-test-slave-d5wc krb5kdc[7767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629839, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:39.512146 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.499446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:47715 (local address 127.21.140.126:33749)
0928 21:10:39.499658 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.499662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.500638 (+ 976us) server_negotiation.cc:407] Connection header received
0928 21:10:39.502011 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.502013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.502061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.502137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:39.503362 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.504163 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.505862 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.506174 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.506260 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.508704 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:39.508707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:39.508709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:39.508735 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:39.510269 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.510793 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.510795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.510797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.510869 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.511186 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.511188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.511189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.511355 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:39.511419 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.511929 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.512027 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 21:10:39.513017 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.499891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33749 (local address 127.21.140.67:47715)
0928 21:10:39.500354 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.500397 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.501783 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.502306 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.502313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.502717 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:39.503201 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.503214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.504317 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.504321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.505733 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.505746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.506317 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.506325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.506467 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.507048 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:39.507065 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:39.508534 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:39.510402 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:39.510407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:39.510419 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:39.510698 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:39.510964 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:39.510966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:39.510968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:39.511108 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:39.511488 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:39.511493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:39.511728 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.511885 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.512153 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260928 21:10:39.514221 8274 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33749
I20260928 21:10:39.514493 8274 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:39.514998 8274 heartbeater.cc:511] Master 127.21.140.126:33749 requested a full tablet report, sending...
I20260928 21:10:39.516083 7812 ts_manager.cc:194] Registered new tserver with Master: 31a1137016d747cca0aa39c55c7d5fdc (127.21.140.67:35433)
I20260928 21:10:39.516780 7812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:47715
I20260928 21:10:39.517887 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:10:39.527895 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.520072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.21.140.126:33749)
0928 21:10:39.520177 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.520180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.520397 (+ 217us) server_negotiation.cc:407] Connection header received
0928 21:10:39.520526 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.520528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.520564 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.520633 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:39.521332 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.521871 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.522875 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.523110 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.523196 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.524757 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:39.524760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:39.524762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:39.524785 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:39.526100 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.526592 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.526595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.526596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.526684 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:39.527096 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:39.527099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:39.527100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:39.527328 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:39.527454 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.527690 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.527774 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:10:39.531021 7812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:10:39.534169 7812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:10:39.549669 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.543314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35433 (local address 127.0.0.1:40890)
0928 21:10:39.544121 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.544179 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.544309 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.545121 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.545127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.545152 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:39.545331 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.545338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.546856 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.546859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.548155 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.548166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.549078 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.549082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.549198 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.549236 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.549391 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.549517 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":34,"thread_start_us":78,"threads_started":1}
I20260928 21:10:39.549680 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.543512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.21.140.67:35433)
0928 21:10:39.544617 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.544630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.544657 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:10:39.544738 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.544743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.544953 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.545112 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:39.545469 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.546716 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.548307 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.548942 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.549076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.549228 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.549427 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.549533 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":146,"threads_started":1}
I20260928 21:10:39.550071 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.543318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:33815 (local address 127.0.0.1:40032)
0928 21:10:39.544543 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.544556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.544641 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.545328 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.545331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.545356 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:39.545496 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.545504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.546640 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.546643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.548152 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.548161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.549468 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.549471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.549677 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.549738 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.549812 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.549886 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":587,"thread_start_us":93,"threads_started":1}
I20260928 21:10:39.550283 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.543643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:39699 (local address 127.0.0.1:47430)
0928 21:10:39.544149 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.544194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.544319 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.545621 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.545624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.545640 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:39.545735 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.545741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.547109 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.547112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.548283 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.548292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.549468 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.549471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.549697 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.549742 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.550042 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.550134 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":99,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260928 21:10:39.550894 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.543555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.21.140.65:33815)
0928 21:10:39.544642 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.544657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.544704 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:10:39.544767 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.544771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.544960 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.545112 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:39.545609 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.546490 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.548279 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.549163 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.549271 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.549401 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.550330 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.550739 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":96,"threads_started":1}
I20260928 21:10:39.551043 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.544106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.21.140.66:39699)
0928 21:10:39.545030 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.545063 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.545086 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:10:39.545153 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.545158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.545337 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.545474 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:39.545866 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.546972 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.548406 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.549177 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.549271 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.549401 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.550619 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.550872 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":133,"threads_started":1}
I20260928 21:10:39.552928 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 72603d81388f469b93a28d73f2541381 (DEFAULT_TABLE table=test-table [id=723ed7b4307740d5b8e0465d957e9114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:39.553196 7938 tablet_service.cc:1467] Processing CreateTablet for tablet 72603d81388f469b93a28d73f2541381 (DEFAULT_TABLE table=test-table [id=723ed7b4307740d5b8e0465d957e9114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:39.553560 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 72603d81388f469b93a28d73f2541381 (DEFAULT_TABLE table=test-table [id=723ed7b4307740d5b8e0465d957e9114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:39.554126 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72603d81388f469b93a28d73f2541381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:39.554224 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72603d81388f469b93a28d73f2541381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:39.554576 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72603d81388f469b93a28d73f2541381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:39.560422 8291 tablet_bootstrap.cc:492] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: Bootstrap starting.
I20260928 21:10:39.562140 8292 tablet_bootstrap.cc:492] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Bootstrap starting.
I20260928 21:10:39.562307 8293 tablet_bootstrap.cc:492] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Bootstrap starting.
I20260928 21:10:39.562794 8291 tablet_bootstrap.cc:654] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:39.563542 8291 log.cc:824] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:39.564383 8293 tablet_bootstrap.cc:654] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:39.564383 8292 tablet_bootstrap.cc:654] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:39.565573 8291 tablet_bootstrap.cc:492] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: No bootstrap required, opened a new log
I20260928 21:10:39.565755 8292 log.cc:824] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:39.565755 8293 log.cc:824] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:39.565805 8291 ts_tablet_manager.cc:1397] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:10:39.567541 8293 tablet_bootstrap.cc:492] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: No bootstrap required, opened a new log
I20260928 21:10:39.567766 8293 ts_tablet_manager.cc:1397] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:10:39.567898 8292 tablet_bootstrap.cc:492] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: No bootstrap required, opened a new log
I20260928 21:10:39.568086 8292 ts_tablet_manager.cc:1397] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:10:39.570408 8291 raft_consensus.cc:348] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.570712 8291 raft_consensus.cc:374] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:39.570801 8291 raft_consensus.cc:729] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9420f256ed4e4992b2b0e130a383a96e, State: Initialized, Role: FOLLOWER
I20260928 21:10:39.570914 8293 raft_consensus.cc:348] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.571228 8293 raft_consensus.cc:374] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:39.571316 8293 raft_consensus.cc:729] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a1137016d747cca0aa39c55c7d5fdc, State: Initialized, Role: FOLLOWER
I20260928 21:10:39.571275 8291 consensus_queue.cc:260] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.571774 8293 consensus_queue.cc:260] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.572083 8004 heartbeater.cc:503] Master 127.21.140.126:33749 was elected leader, sending a full tablet report...
I20260928 21:10:39.572397 8291 ts_tablet_manager.cc:1428] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260928 21:10:39.572820 8293 ts_tablet_manager.cc:1428] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:10:39.572845 8292 raft_consensus.cc:348] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.573251 8274 heartbeater.cc:503] Master 127.21.140.126:33749 was elected leader, sending a full tablet report...
I20260928 21:10:39.573273 8292 raft_consensus.cc:374] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:39.573335 8292 raft_consensus.cc:729] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebee74c17fe04c4b8f7630175c457abb, State: Initialized, Role: FOLLOWER
I20260928 21:10:39.573797 8292 consensus_queue.cc:260] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.574597 8140 heartbeater.cc:503] Master 127.21.140.126:33749 was elected leader, sending a full tablet report...
I20260928 21:10:39.574841 8292 ts_tablet_manager.cc:1428] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260928 21:10:39.748651 8275 tablet.cc:2369] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:10:39.772732 8141 tablet.cc:2369] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:10:39.789150 8005 tablet.cc:2369] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:10:39.795698 8299 raft_consensus.cc:482] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:39.795940 8299 raft_consensus.cc:504] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.797474 8299 leader_election.cc:288] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31a1137016d747cca0aa39c55c7d5fdc (127.21.140.67:35433), 9420f256ed4e4992b2b0e130a383a96e (127.21.140.65:33815)
I20260928 21:10:39.802433 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.797937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:37159 (local address 127.21.140.65:33815)
0928 21:10:39.798105 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.798112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.798578 (+ 466us) server_negotiation.cc:407] Connection header received
0928 21:10:39.798734 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.798739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.798808 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.798864 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:39.799361 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.800003 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.801225 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.801744 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.801863 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.801915 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.802207 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.802284 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 21:10:39.802520 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.797704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:33815 (local address 127.21.140.66:37159)
0928 21:10:39.798419 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.798436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.798568 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.799053 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.799059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.799081 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:39.799236 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.799247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.800121 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.800126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.801105 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.801114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.801887 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.801894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.802046 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.802090 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.802178 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.802279 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":388,"thread_start_us":73,"threads_started":1}
I20260928 21:10:39.802982 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.797704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:35433 (local address 127.21.140.66:55255)
0928 21:10:39.798087 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.798103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:39.798229 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:39.798642 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:39.798645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:39.798663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:39.798781 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.798786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.799751 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.799757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.801337 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:39.801351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.802372 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.802376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:39.802473 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.802500 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:10:39.802768 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 21:10:39.802841 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":94,"threads_started":1}
I20260928 21:10:39.803093 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.797895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:55255 (local address 127.21.140.67:35433)
0928 21:10:39.798062 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.798067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.798208 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:10:39.798408 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.798414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.798464 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.798595 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:39.798886 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.799623 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.801478 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.802103 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.802237 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.802282 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.802820 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.802927 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:10:39.803447 7958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72603d81388f469b93a28d73f2541381" candidate_uuid: "ebee74c17fe04c4b8f7630175c457abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9420f256ed4e4992b2b0e130a383a96e" is_pre_election: true
I20260928 21:10:39.803794 7958 raft_consensus.cc:2454] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebee74c17fe04c4b8f7630175c457abb in term 0.
I20260928 21:10:39.803758 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72603d81388f469b93a28d73f2541381" candidate_uuid: "ebee74c17fe04c4b8f7630175c457abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1137016d747cca0aa39c55c7d5fdc" is_pre_election: true
I20260928 21:10:39.804050 8228 raft_consensus.cc:2454] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebee74c17fe04c4b8f7630175c457abb in term 0.
I20260928 21:10:39.804317 8027 leader_election.cc:302] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9420f256ed4e4992b2b0e130a383a96e, ebee74c17fe04c4b8f7630175c457abb; no voters:
I20260928 21:10:39.804636 8299 raft_consensus.cc:2781] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:10:39.804723 8299 raft_consensus.cc:482] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:39.804766 8299 raft_consensus.cc:3037] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:39.805783 8299 raft_consensus.cc:504] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.806181 8299 leader_election.cc:288] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [CANDIDATE]: Term 1 election: Requested vote from peers 31a1137016d747cca0aa39c55c7d5fdc (127.21.140.67:35433), 9420f256ed4e4992b2b0e130a383a96e (127.21.140.65:33815)
I20260928 21:10:39.806557 7958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72603d81388f469b93a28d73f2541381" candidate_uuid: "ebee74c17fe04c4b8f7630175c457abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9420f256ed4e4992b2b0e130a383a96e"
I20260928 21:10:39.806555 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72603d81388f469b93a28d73f2541381" candidate_uuid: "ebee74c17fe04c4b8f7630175c457abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a1137016d747cca0aa39c55c7d5fdc"
I20260928 21:10:39.806689 7958 raft_consensus.cc:3037] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:39.806707 8228 raft_consensus.cc:3037] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:39.807714 7958 raft_consensus.cc:2454] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebee74c17fe04c4b8f7630175c457abb in term 1.
I20260928 21:10:39.808025 8027 leader_election.cc:302] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9420f256ed4e4992b2b0e130a383a96e, ebee74c17fe04c4b8f7630175c457abb; no voters:
I20260928 21:10:39.808215 8299 raft_consensus.cc:2781] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:39.808346 8228 raft_consensus.cc:2454] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebee74c17fe04c4b8f7630175c457abb in term 1.
I20260928 21:10:39.808467 8299 raft_consensus.cc:686] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 1 LEADER]: Becoming Leader. State: Replica: ebee74c17fe04c4b8f7630175c457abb, State: Running, Role: LEADER
I20260928 21:10:39.808825 8299 consensus_queue.cc:237] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } }
I20260928 21:10:39.812346 7811 catalog_manager.cc:5495] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb reported cstate change: term changed from 0 to 1, leader changed from <none> to ebee74c17fe04c4b8f7630175c457abb (127.21.140.66). New cstate: current_term: 1 leader_uuid: "ebee74c17fe04c4b8f7630175c457abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebee74c17fe04c4b8f7630175c457abb" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 39699 } health_report { overall_health: HEALTHY } } }
I20260928 21:10:39.845608 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.840307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.21.140.66:39699)
0928 21:10:39.840431 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:39.840453 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:39.840469 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:10:39.840678 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:39.840690 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:39.840768 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:39.840878 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:10:39.841261 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.842157 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.843451 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:39.844004 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:39.844131 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:39.844485 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:10:39.845099 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:10:39.845278 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:39.845414 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 21:10:39.845455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:10:39.852596 8228 raft_consensus.cc:1261] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 1 FOLLOWER]: Refusing update from remote peer ebee74c17fe04c4b8f7630175c457abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:39.852596 7958 raft_consensus.cc:1261] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 1 FOLLOWER]: Refusing update from remote peer ebee74c17fe04c4b8f7630175c457abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:39.853344 8302 consensus_queue.cc:1035] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a1137016d747cca0aa39c55c7d5fdc" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:39.853624 8299 consensus_queue.cc:1035] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9420f256ed4e4992b2b0e130a383a96e" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:39.861466 8305 mvcc.cc:205] Tried to move back new op lower bound from 7334419824028286976 to 7334419823859417088. Current Snapshot: MvccSnapshot[applied={T|T < 7334419824028286976}]
I20260928 21:10:39.863191 8307 mvcc.cc:205] Tried to move back new op lower bound from 7334419824028286976 to 7334419823859417088. Current Snapshot: MvccSnapshot[applied={T|T < 7334419824028286976}]
I20260928 21:10:39.867754 8306 mvcc.cc:205] Tried to move back new op lower bound from 7334419824028286976 to 7334419823859417088. Current Snapshot: MvccSnapshot[applied={T|T < 7334419824028286976 or (T in {7334419824028286976})}]
I20260928 21:10:39.874867 7812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:39.875051 7812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:39.877416 7812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d96fce78026d4067ac7705984d58f8ab: Sending DeleteTablet for 3 replicas of tablet 72603d81388f469b93a28d73f2541381
I20260928 21:10:39.878316 7938 tablet_service.cc:1514] Processing DeleteTablet for tablet 72603d81388f469b93a28d73f2541381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:39 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:40032
I20260928 21:10:39.878472 8073 tablet_service.cc:1514] Processing DeleteTablet for tablet 72603d81388f469b93a28d73f2541381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:39 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:47430
I20260928 21:10:39.878499 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 72603d81388f469b93a28d73f2541381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:39 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:40890
I20260928 21:10:39.879007 8315 tablet_replica.cc:330] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e: stopping tablet replica
I20260928 21:10:39.879380 8315 raft_consensus.cc:2229] T 72603d81388f469b93a28d73f2541381 P 9420f256ed4e4992b2b0e130a383a96e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:39.879487 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7875
I20260928 21:10:39.881503 8316 tablet_replica.cc:330] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: stopping tablet replica
I20260928 21:10:39.881810 8316 raft_consensus.cc:2229] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:39.882151 8317 tablet_replica.cc:330] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: stopping tablet replica
I20260928 21:10:39.882196 8316 raft_consensus.cc:2258] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:39.882483 8317 raft_consensus.cc:2229] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:10:39.882926 8317 raft_consensus.cc:2258] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:39.884274 8316 ts_tablet_manager.cc:1905] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:39.884413 8317 ts_tablet_manager.cc:1905] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:39.887171 8316 ts_tablet_manager.cc:1918] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:39.887360 8316 log.cc:1196] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/72603d81388f469b93a28d73f2541381
I20260928 21:10:39.887768 8316 ts_tablet_manager.cc:1939] T 72603d81388f469b93a28d73f2541381 P 31a1137016d747cca0aa39c55c7d5fdc: Deleting consensus metadata
I20260928 21:10:39.887940 8317 ts_tablet_manager.cc:1918] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:39.888111 8317 log.cc:1196] T 72603d81388f469b93a28d73f2541381 P ebee74c17fe04c4b8f7630175c457abb: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/72603d81388f469b93a28d73f2541381
W20260928 21:10:39.888387 7798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:33815 (error 108)
I20260928 21:10:39.888583 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8009
I20260928 21:10:39.890249 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.889858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:33815 (local address 127.0.0.1:40034)
0928 21:10:39.890025 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.890109 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:10:39.890455 7798 catalog_manager.cc:4584] TS 9420f256ed4e4992b2b0e130a383a96e (127.21.140.65:33815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72603d81388f469b93a28d73f2541381: Network error: Client connection negotiation failed: client connection to 127.21.140.65:33815: connect: Connection refused (error 111)
I20260928 21:10:39.891108 7796 catalog_manager.cc:4841] TS 31a1137016d747cca0aa39c55c7d5fdc (127.21.140.67:35433): tablet 72603d81388f469b93a28d73f2541381 (table test-table [id=723ed7b4307740d5b8e0465d957e9114]) successfully deleted
I20260928 21:10:39.895449 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8145
I20260928 21:10:39.896265 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:39.895642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:39699 (local address 127.0.0.1:47450)
0928 21:10:39.896100 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:39.896162 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.66:39699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
I20260928 21:10:39.901145 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 7780
2026-09-28T21:10:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2532 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T21:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:39Z Disabled control of system clock
I20260928 21:10:39.932204 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33099
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:33099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:10:40.044265 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:40.044585 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:40.044623 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:40.048919 8323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:40.048990 8323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:40.049006 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:40.049024 8323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:40.049039 8323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:40.054210 8323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:33099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:40.055400 8323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:40.056859 8323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:40.065662 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.065762 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.065662 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:40.065956 8323 server_base.cc:1034] running on GCE node
I20260928 21:10:40.066628 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:40.067742 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:40.068986 8323 hybrid_clock.cc:648] HybridClock initialized: now 1790629840068955 us; error 46 us; skew 500 ppm
I20260928 21:10:40.070958 8323 webserver.cc:460] Webserver started at http://127.21.140.126:34593/ using document root <none> and password file <none>
I20260928 21:10:40.071573 8323 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:40.071667 8323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:40.071904 8323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:40.074034 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "9dd5680531254af7956a7da763807092"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "e2a0c00feb08cbca246aaf91004f9506"
server_key_iv: "b1d34985c1e30100fae9a9f39a73532e"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.074648 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "9dd5680531254af7956a7da763807092"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "e2a0c00feb08cbca246aaf91004f9506"
server_key_iv: "b1d34985c1e30100fae9a9f39a73532e"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.078624 8323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:10:40.081095 8337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.082549 8323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:40.082693 8323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "9dd5680531254af7956a7da763807092"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "e2a0c00feb08cbca246aaf91004f9506"
server_key_iv: "b1d34985c1e30100fae9a9f39a73532e"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.082815 8323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:40.097110 8323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:40.097813 8323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:40.098073 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:40.105383 8323 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:33099
I20260928 21:10:40.105399 8389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:33099 every 8 connection(s)
I20260928 21:10:40.106396 8323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:40.108302 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8323
I20260928 21:10:40.108469 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:40.108832 22065 external_mini_cluster.cc:1428] Setting key c88aea25c122e1e00e4085bb2a65bf2c
I20260928 21:10:40.110423 8390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:40.116392 8390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092: Bootstrap starting.
I20260928 21:10:40.118865 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.110470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.21.140.126:33099)
0928 21:10:40.111194 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.111211 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.111249 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:10:40.111991 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.112007 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.112424 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.112810 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:40.113271 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.114424 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.115407 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.115923 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.116122 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.117149 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:40.117154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:40.117166 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:40.117543 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:40.117597 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:40.117616 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:40.117730 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.117858 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.118144 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":152,"threads_started":1}
I20260928 21:10:40.118914 8390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:40.119879 8390 log.cc:824] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:40.122081 8390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092: No bootstrap required, opened a new log
I20260928 21:10:40.125922 8390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd5680531254af7956a7da763807092" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33099 } }
I20260928 21:10:40.126214 8390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:40.126266 8390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd5680531254af7956a7da763807092, State: Initialized, Role: FOLLOWER
I20260928 21:10:40.126665 8390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd5680531254af7956a7da763807092" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33099 } }
I20260928 21:10:40.126785 8390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:40.126828 8390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:40.126904 8390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:40.127827 8390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd5680531254af7956a7da763807092" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33099 } }
I20260928 21:10:40.128139 8390 leader_election.cc:302] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dd5680531254af7956a7da763807092; no voters:
I20260928 21:10:40.128412 8390 leader_election.cc:288] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:40.128557 8395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:40.128786 8395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [term 1 LEADER]: Becoming Leader. State: Replica: 9dd5680531254af7956a7da763807092, State: Running, Role: LEADER
I20260928 21:10:40.129110 8395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd5680531254af7956a7da763807092" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33099 } }
I20260928 21:10:40.129809 8390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:40.130851 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dd5680531254af7956a7da763807092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd5680531254af7956a7da763807092" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33099 } } }
I20260928 21:10:40.130832 8397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dd5680531254af7956a7da763807092. Latest consensus state: current_term: 1 leader_uuid: "9dd5680531254af7956a7da763807092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dd5680531254af7956a7da763807092" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33099 } } }
I20260928 21:10:40.131099 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:40.131110 8397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:40.131857 8402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:40.134975 8402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:40.140899 8402 catalog_manager.cc:1347] Generated new cluster ID: f6d23dc186dd46a6b1afb9331ab89528
I20260928 21:10:40.141006 8402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:40.158833 8402 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:40.160169 8402 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:40.170832 8402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092: Generated new TSK 0
I20260928 21:10:40.171736 8402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:40.184463 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33099
--builtin_ntp_servers=127.21.140.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:10:40.302908 8415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:40.303160 8415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:40.303200 8415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:40.307329 8415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:40.307390 8415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:40.307448 8415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:40.312342 8415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:40.313392 8415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:40.314821 8415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:40.324515 8423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:40.324584 8415 server_base.cc:1034] running on GCE node
W20260928 21:10:40.324591 8421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.324609 8420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:40.325189 8415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:40.325850 8415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:40.327049 8415 hybrid_clock.cc:648] HybridClock initialized: now 1790629840327008 us; error 55 us; skew 500 ppm
I20260928 21:10:40.328955 8415 webserver.cc:460] Webserver started at http://127.21.140.65:36007/ using document root <none> and password file <none>
I20260928 21:10:40.329576 8415 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:40.329733 8415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:40.330010 8415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:40.332228 8415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "6127163e3d5743eaa63ee850a4d21b00"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "38ab64e65c80610ba07182edbd9fee3a"
server_key_iv: "5d733ab8afe1bd653a057b6f44befe73"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.332798 8415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "6127163e3d5743eaa63ee850a4d21b00"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "38ab64e65c80610ba07182edbd9fee3a"
server_key_iv: "5d733ab8afe1bd653a057b6f44befe73"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.336624 8415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:40.339215 8429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.340505 8415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:10:40.340689 8415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "6127163e3d5743eaa63ee850a4d21b00"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "38ab64e65c80610ba07182edbd9fee3a"
server_key_iv: "5d733ab8afe1bd653a057b6f44befe73"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.340812 8415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:40.358258 8415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:40.358999 8415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:40.359206 8415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:40.359826 8415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:40.360819 8415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:40.360900 8415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.360991 8415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:40.361042 8415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.370164 8415 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:44171
I20260928 21:10:40.370190 8542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:44171 every 8 connection(s)
I20260928 21:10:40.371132 8415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:40.371445 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8415
I20260928 21:10:40.371582 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:10:40.371937 22065 external_mini_cluster.cc:1428] Setting key 12814ecc76aa4b218a5ba8c797b5c410
I20260928 21:10:40.374239 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33099
--builtin_ntp_servers=127.21.140.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:10:40.381808 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.373056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:57559 (local address 127.21.140.126:33099)
0928 21:10:40.373206 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.373211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.374630 (+ 1419us) server_negotiation.cc:407] Connection header received
0928 21:10:40.376230 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.376234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.376298 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.376430 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:40.377342 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.377896 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.379289 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.379754 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.379907 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.381162 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:40.381169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:40.381173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:40.381194 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:40.381223 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:40.381238 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:40.381331 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.381489 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.381579 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:10:40.382711 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.373597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33099 (local address 127.21.140.65:57559)
0928 21:10:40.374449 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.374507 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.376005 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.376600 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.376608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.376700 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:40.377198 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.377211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.378059 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.378063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.379140 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.379152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.379934 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.379937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.380095 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.380854 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:40.380856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:40.380900 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:40.380980 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:40.381336 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:40.381344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:40.381349 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:10:40.381463 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:10:40.381722 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":138,"threads_started":1}
I20260928 21:10:40.384159 8543 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33099
I20260928 21:10:40.384420 8543 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:40.384963 8543 heartbeater.cc:511] Master 127.21.140.126:33099 requested a full tablet report, sending...
I20260928 21:10:40.386549 8354 ts_manager.cc:194] Registered new tserver with Master: 6127163e3d5743eaa63ee850a4d21b00 (127.21.140.65:44171)
I20260928 21:10:40.387743 8354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.65:57559
W20260928 21:10:40.491952 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:40.492224 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:40.492264 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:40.497020 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:40.497087 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:40.497170 8547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:40.502182 8547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:40.503274 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:40.504673 8547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:40.514680 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.514668 8555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.514674 8552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:40.515077 8547 server_base.cc:1034] running on GCE node
I20260928 21:10:40.515518 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:40.516207 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:40.517406 8547 hybrid_clock.cc:648] HybridClock initialized: now 1790629840517374 us; error 46 us; skew 500 ppm
I20260928 21:10:40.519407 8547 webserver.cc:460] Webserver started at http://127.21.140.66:41025/ using document root <none> and password file <none>
I20260928 21:10:40.519999 8547 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:40.520094 8547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:40.520335 8547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:40.522473 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "c028efc2e54d4d4195615316786b660e"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "00414a7619917b06914ac622081761a8"
server_key_iv: "fd3eb49d699f4c636b2fe9a766c5aa00"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.523061 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "c028efc2e54d4d4195615316786b660e"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "00414a7619917b06914ac622081761a8"
server_key_iv: "fd3eb49d699f4c636b2fe9a766c5aa00"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.526785 8547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:10:40.529335 8561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.530762 8547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:40.530905 8547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "c028efc2e54d4d4195615316786b660e"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "00414a7619917b06914ac622081761a8"
server_key_iv: "fd3eb49d699f4c636b2fe9a766c5aa00"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.531025 8547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:40.546911 8547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:40.547641 8547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:40.547863 8547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:40.548470 8547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:40.549420 8547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:40.549468 8547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.549538 8547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:40.549580 8547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.558902 8547 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:41455
I20260928 21:10:40.558990 8674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:41455 every 8 connection(s)
I20260928 21:10:40.559849 8547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:40.561398 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8547
I20260928 21:10:40.561494 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:10:40.561816 22065 external_mini_cluster.cc:1428] Setting key 2a6b605c33bb512cbb60ec08223d4b82
I20260928 21:10:40.563967 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33099
--builtin_ntp_servers=127.21.140.84:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:10:40.571805 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.561879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:44027 (local address 127.21.140.126:33099)
0928 21:10:40.562057 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.562062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.563731 (+ 1669us) server_negotiation.cc:407] Connection header received
0928 21:10:40.565819 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.565825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.565904 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.566058 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:40.566973 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.567545 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.569530 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.569753 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.569852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.571264 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:40.571267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:40.571269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:40.571283 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:40.571303 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:40.571310 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:40.571374 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.571621 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.571687 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:10:40.573125 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.562366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33099 (local address 127.21.140.66:44027)
0928 21:10:40.563552 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.563607 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.565525 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.566220 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.566227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.566316 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:40.566827 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.566841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.567694 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.567701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.569360 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.569376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.569934 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.569941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.570209 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.570914 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:40.570918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:40.570982 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:40.571081 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:40.571407 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:40.571419 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:40.571429 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:10:40.571594 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 21:10:40.571939 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":143,"threads_started":1}
I20260928 21:10:40.574590 8675 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33099
I20260928 21:10:40.574870 8675 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:40.575456 8675 heartbeater.cc:511] Master 127.21.140.126:33099 requested a full tablet report, sending...
I20260928 21:10:40.576602 8354 ts_manager.cc:194] Registered new tserver with Master: c028efc2e54d4d4195615316786b660e (127.21.140.66:41455)
I20260928 21:10:40.577332 8354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.66:44027
W20260928 21:10:40.682663 8679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:40.682920 8679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:40.682960 8679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:40.687136 8679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:40.687227 8679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:40.687314 8679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:40.692418 8679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:40.693557 8679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:40.694983 8679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:40.705044 8685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.705035 8687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:40.705034 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:40.705169 8679 server_base.cc:1034] running on GCE node
I20260928 21:10:40.705910 8679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:40.706661 8679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:40.707851 8679 hybrid_clock.cc:648] HybridClock initialized: now 1790629840707811 us; error 48 us; skew 500 ppm
I20260928 21:10:40.709954 8679 webserver.cc:460] Webserver started at http://127.21.140.67:33199/ using document root <none> and password file <none>
I20260928 21:10:40.710734 8679 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:40.710806 8679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:40.711057 8679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:40.713253 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "14e3fcaf575cfd805b4500e8d2ea4bec"
server_key_iv: "0c8ccab8ae762fac5f87e6178909563d"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.713848 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "14e3fcaf575cfd805b4500e8d2ea4bec"
server_key_iv: "0c8ccab8ae762fac5f87e6178909563d"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.717721 8679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 21:10:40.720398 8693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.721788 8679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:10:40.721931 8679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401"
format_stamp: "Formatted at 2026-09-28 21:10:40 on dist-test-slave-d5wc"
server_key: "14e3fcaf575cfd805b4500e8d2ea4bec"
server_key_iv: "0c8ccab8ae762fac5f87e6178909563d"
server_key_version: "encryptionkey@0"
I20260928 21:10:40.722128 8679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:40.732975 8679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:40.733668 8679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:40.733866 8679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:40.734615 8679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:40.735675 8679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:40.735745 8679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.735812 8679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:40.735867 8679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:40.745191 8679 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:33881
I20260928 21:10:40.745220 8806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:33881 every 8 connection(s)
I20260928 21:10:40.746254 8679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:10:40.751506 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8679
I20260928 21:10:40.751621 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:10:40.751909 22065 external_mini_cluster.cc:1428] Setting key 3ec9d6857d76d7aa716f2ac2f8c061c6
I20260928 21:10:40.756310 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.748172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:50667 (local address 127.21.140.126:33099)
0928 21:10:40.748306 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.748310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.749335 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 21:10:40.750844 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.750848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.750910 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.751027 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:40.752285 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.753028 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.754453 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.754679 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.754770 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.755898 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:40.755901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:40.755903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:40.755915 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:40.755934 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:40.755941 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:40.756007 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.756150 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.756201 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:10:40.757164 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.748459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33099 (local address 127.21.140.67:50667)
0928 21:10:40.749167 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.749208 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.750629 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.751198 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.751205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.751352 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:40.752098 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.752118 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.753201 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.753205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.754262 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.754273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.754831 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.754834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.754984 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.755635 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:40.755637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:40.755678 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:40.755743 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:40.756024 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:40.756030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:40.756034 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:10:40.756129 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:10:40.756344 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":114,"threads_started":1}
I20260928 21:10:40.758371 8807 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33099
I20260928 21:10:40.758679 8807 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:40.759208 8807 heartbeater.cc:511] Master 127.21.140.126:33099 requested a full tablet report, sending...
I20260928 21:10:40.760281 8354 ts_manager.cc:194] Registered new tserver with Master: e5b1b7ecb4a04e92b6a6c5cb1a611401 (127.21.140.67:33881)
I20260928 21:10:40.760881 8354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.67:50667
I20260928 21:10:40.766314 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:10:40.771915 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.768254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.21.140.126:33099)
0928 21:10:40.768378 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.768381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.768533 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:10:40.768621 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.768623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.768660 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.768726 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:40.768972 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.769465 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.770391 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.770610 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.770703 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.771513 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:40.771516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:40.771518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:40.771529 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:40.771547 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:40.771555 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:40.771663 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.771776 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.771807 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:10:40.775111 8354 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:10:40.778162 8354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:10:40.792946 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.787093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:41455 (local address 127.0.0.1:47422)
0928 21:10:40.787483 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.787534 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.787684 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.788523 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.788530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.788555 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:40.788722 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.788729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.789912 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.789919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.791230 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.791240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.792351 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.792357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.792466 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.792502 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:10:40.792624 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:10:40.792772 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":6,"spinlock_wait_cycles":7936,"thread_start_us":104,"threads_started":1}
I20260928 21:10:40.793207 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.787509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.21.140.67:33881)
0928 21:10:40.788177 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.788193 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.788216 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:10:40.788290 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.788295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.788486 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.788667 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:40.789114 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.790278 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.791711 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.792574 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.792711 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.792890 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.792953 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.793052 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":169,"threads_started":1}
I20260928 21:10:40.793255 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.786828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.21.140.65:44171)
0928 21:10:40.788168 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.788179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.788198 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:10:40.788261 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.788265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.788518 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.788662 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:40.789079 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.790321 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.791647 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.792307 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.792412 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.792600 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.793012 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.793098 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":119,"threads_started":1}
I20260928 21:10:40.793519 8821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.786666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:44171 (local address 127.0.0.1:58766)
0928 21:10:40.787473 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.787535 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.787685 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.788811 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.788817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.788834 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:40.788943 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.788949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.790451 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.790454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.791506 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.791526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.792613 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.792619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.792718 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.792740 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:10:40.792809 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:10:40.792882 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":6,"spinlock_wait_cycles":8832,"thread_start_us":99,"threads_started":1}
I20260928 21:10:40.793646 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.786827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:33881 (local address 127.0.0.1:51556)
0928 21:10:40.787849 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.787861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.787933 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.788831 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.788834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.788857 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:40.788961 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.788965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.790414 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.790418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.791579 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.791589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.792737 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.792740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:40.792817 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.792837 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:10:40.792899 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:10:40.792959 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":48,"spinlock_wait_cycles":768,"thread_start_us":115,"threads_started":1}
I20260928 21:10:40.793646 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.787502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.21.140.66:41455)
0928 21:10:40.787996 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.788012 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.788039 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:10:40.788121 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.788127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.788345 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.788871 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:40.789030 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.789786 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.791389 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.792204 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.792312 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:40.792522 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:40.793328 (+ 806us) server_negotiation.cc:299] Negotiation successful
0928 21:10:40.793435 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":100,"threads_started":1}
I20260928 21:10:40.796728 8609 tablet_service.cc:1467] Processing CreateTablet for tablet 5047e2feec5a4ebd8db74e6d11eead30 (DEFAULT_TABLE table=test-table [id=356fb67e41c94df5a8658c91206a34a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:40.796728 8477 tablet_service.cc:1467] Processing CreateTablet for tablet 5047e2feec5a4ebd8db74e6d11eead30 (DEFAULT_TABLE table=test-table [id=356fb67e41c94df5a8658c91206a34a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:40.797911 8477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047e2feec5a4ebd8db74e6d11eead30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:40.797912 8609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047e2feec5a4ebd8db74e6d11eead30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:40.800050 8741 tablet_service.cc:1467] Processing CreateTablet for tablet 5047e2feec5a4ebd8db74e6d11eead30 (DEFAULT_TABLE table=test-table [id=356fb67e41c94df5a8658c91206a34a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:40.800971 8741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047e2feec5a4ebd8db74e6d11eead30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:40.805575 8825 tablet_bootstrap.cc:492] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: Bootstrap starting.
I20260928 21:10:40.806044 8824 tablet_bootstrap.cc:492] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Bootstrap starting.
I20260928 21:10:40.807324 8826 tablet_bootstrap.cc:492] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Bootstrap starting.
I20260928 21:10:40.808337 8824 tablet_bootstrap.cc:654] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:40.808624 8825 tablet_bootstrap.cc:654] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:40.809360 8824 log.cc:824] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:40.809420 8826 tablet_bootstrap.cc:654] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:40.809489 8825 log.cc:824] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:40.810302 8826 log.cc:824] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:40.811623 8824 tablet_bootstrap.cc:492] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: No bootstrap required, opened a new log
I20260928 21:10:40.811827 8824 ts_tablet_manager.cc:1397] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:10:40.812052 8825 tablet_bootstrap.cc:492] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: No bootstrap required, opened a new log
I20260928 21:10:40.812251 8825 ts_tablet_manager.cc:1397] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:10:40.812248 8826 tablet_bootstrap.cc:492] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: No bootstrap required, opened a new log
I20260928 21:10:40.812431 8826 ts_tablet_manager.cc:1397] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:10:40.815372 8825 raft_consensus.cc:348] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:40.815776 8825 raft_consensus.cc:374] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:40.815887 8825 raft_consensus.cc:729] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6127163e3d5743eaa63ee850a4d21b00, State: Initialized, Role: FOLLOWER
I20260928 21:10:40.816216 8824 raft_consensus.cc:348] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:40.816457 8825 consensus_queue.cc:260] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:40.816558 8824 raft_consensus.cc:374] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:40.816644 8824 raft_consensus.cc:729] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c028efc2e54d4d4195615316786b660e, State: Initialized, Role: FOLLOWER
I20260928 21:10:40.816566 8826 raft_consensus.cc:348] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:40.816808 8826 raft_consensus.cc:374] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:40.816864 8826 raft_consensus.cc:729] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b1b7ecb4a04e92b6a6c5cb1a611401, State: Initialized, Role: FOLLOWER
I20260928 21:10:40.817154 8824 consensus_queue.cc:260] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:40.817211 8826 consensus_queue.cc:260] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:40.817631 8825 ts_tablet_manager.cc:1428] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:10:40.817869 8807 heartbeater.cc:503] Master 127.21.140.126:33099 was elected leader, sending a full tablet report...
I20260928 21:10:40.818104 8675 heartbeater.cc:503] Master 127.21.140.126:33099 was elected leader, sending a full tablet report...
I20260928 21:10:40.818140 8826 ts_tablet_manager.cc:1428] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:10:40.818990 8543 heartbeater.cc:503] Master 127.21.140.126:33099 was elected leader, sending a full tablet report...
I20260928 21:10:40.819015 8824 ts_tablet_manager.cc:1428] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260928 21:10:40.872844 8545 tablet.cc:2369] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:10:40.996182 8831 raft_consensus.cc:482] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:40.996402 8831 raft_consensus.cc:504] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
W20260928 21:10:40.997237 8808 tablet.cc:2369] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:10:40.997818 8831 leader_election.cc:288] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6127163e3d5743eaa63ee850a4d21b00 (127.21.140.65:44171), c028efc2e54d4d4195615316786b660e (127.21.140.66:41455)
I20260928 21:10:41.001531 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.998196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:34537 (local address 127.21.140.66:41455)
0928 21:10:40.998331 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.998335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.998351 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:10:40.998518 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.998521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.998568 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.998658 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:40.998953 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.999459 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.000529 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:41.000942 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.001085 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:41.001148 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:41.001331 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:10:41.001410 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:10:41.001533 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.998066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:41455 (local address 127.21.140.67:34537)
0928 21:10:40.998218 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.998237 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.998356 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.998673 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.998676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.998694 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:40.998828 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.998834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.999584 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.999590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:41.000373 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:41.000381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:41.001085 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.001089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:41.001193 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:41.001221 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:10:41.001313 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:10:41.001390 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":1,"spinlock_wait_cycles":8704}
I20260928 21:10:41.001703 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.998066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:44171 (local address 127.21.140.67:55667)
0928 21:10:40.998580 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:40.998592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:40.998734 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:40.998997 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:40.998999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:40.999014 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:40.999137 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:40.999142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.999855 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:40.999859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:41.000618 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:41.000626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:41.001267 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.001271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:41.001367 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:41.001393 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:10:41.001482 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:10:41.001558 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":133,"thread_start_us":132,"threads_started":1}
I20260928 21:10:41.001686 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:40.998187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:55667 (local address 127.21.140.65:44171)
0928 21:10:40.998332 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:40.998339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:40.998679 (+ 340us) server_negotiation.cc:407] Connection header received
0928 21:10:40.998840 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:40.998843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:40.998894 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:40.998985 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:40.999244 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:40.999738 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.000746 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:41.001141 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.001274 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:41.001318 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:41.001499 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:10:41.001570 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:10:41.002461 8629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047e2feec5a4ebd8db74e6d11eead30" candidate_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c028efc2e54d4d4195615316786b660e" is_pre_election: true
I20260928 21:10:41.002452 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047e2feec5a4ebd8db74e6d11eead30" candidate_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6127163e3d5743eaa63ee850a4d21b00" is_pre_election: true
I20260928 21:10:41.002790 8497 raft_consensus.cc:2454] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5b1b7ecb4a04e92b6a6c5cb1a611401 in term 0.
I20260928 21:10:41.002790 8629 raft_consensus.cc:2454] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5b1b7ecb4a04e92b6a6c5cb1a611401 in term 0.
I20260928 21:10:41.003351 8696 leader_election.cc:302] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6127163e3d5743eaa63ee850a4d21b00, c028efc2e54d4d4195615316786b660e, e5b1b7ecb4a04e92b6a6c5cb1a611401; no voters:
I20260928 21:10:41.003692 8831 raft_consensus.cc:2781] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:10:41.003791 8831 raft_consensus.cc:482] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:41.003851 8831 raft_consensus.cc:3037] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:41.004899 8831 raft_consensus.cc:504] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:41.005259 8831 leader_election.cc:288] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [CANDIDATE]: Term 1 election: Requested vote from peers 6127163e3d5743eaa63ee850a4d21b00 (127.21.140.65:44171), c028efc2e54d4d4195615316786b660e (127.21.140.66:41455)
I20260928 21:10:41.005755 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047e2feec5a4ebd8db74e6d11eead30" candidate_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6127163e3d5743eaa63ee850a4d21b00"
I20260928 21:10:41.005813 8629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5047e2feec5a4ebd8db74e6d11eead30" candidate_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c028efc2e54d4d4195615316786b660e"
I20260928 21:10:41.005942 8629 raft_consensus.cc:3037] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:41.005954 8497 raft_consensus.cc:3037] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:41.007081 8629 raft_consensus.cc:2454] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5b1b7ecb4a04e92b6a6c5cb1a611401 in term 1.
I20260928 21:10:41.007174 8497 raft_consensus.cc:2454] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5b1b7ecb4a04e92b6a6c5cb1a611401 in term 1.
I20260928 21:10:41.007433 8697 leader_election.cc:302] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c028efc2e54d4d4195615316786b660e, e5b1b7ecb4a04e92b6a6c5cb1a611401; no voters:
I20260928 21:10:41.007695 8831 raft_consensus.cc:2781] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:41.007990 8831 raft_consensus.cc:686] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 1 LEADER]: Becoming Leader. State: Replica: e5b1b7ecb4a04e92b6a6c5cb1a611401, State: Running, Role: LEADER
I20260928 21:10:41.008352 8831 consensus_queue.cc:237] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } }
I20260928 21:10:41.011822 8353 catalog_manager.cc:5495] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5b1b7ecb4a04e92b6a6c5cb1a611401 (127.21.140.67). New cstate: current_term: 1 leader_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5b1b7ecb4a04e92b6a6c5cb1a611401" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 33881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 } health_report { overall_health: UNKNOWN } } }
I20260928 21:10:41.031554 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:41.027613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.21.140.67:33881)
0928 21:10:41.027743 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:41.027746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:41.027761 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:10:41.027862 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:41.027865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:41.027921 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:41.028013 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:10:41.028249 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:41.028812 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.029872 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:41.030187 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:41.030303 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:41.030643 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:10:41.031082 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:10:41.031198 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:41.031380 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 21:10:41.031426 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:10:41.038137 8497 raft_consensus.cc:1261] T 5047e2feec5a4ebd8db74e6d11eead30 P 6127163e3d5743eaa63ee850a4d21b00 [term 1 FOLLOWER]: Refusing update from remote peer e5b1b7ecb4a04e92b6a6c5cb1a611401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:41.038138 8629 raft_consensus.cc:1261] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 1 FOLLOWER]: Refusing update from remote peer e5b1b7ecb4a04e92b6a6c5cb1a611401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:41.038954 8834 consensus_queue.cc:1035] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c028efc2e54d4d4195615316786b660e" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:41.039137 8831 consensus_queue.cc:1035] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6127163e3d5743eaa63ee850a4d21b00" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:41.047345 8837 mvcc.cc:205] Tried to move back new op lower bound from 7334419828883435520 to 7334419828772896768. Current Snapshot: MvccSnapshot[applied={T|T < 7334419828883435520}]
I20260928 21:10:41.048702 8838 mvcc.cc:205] Tried to move back new op lower bound from 7334419828883435520 to 7334419828772896768. Current Snapshot: MvccSnapshot[applied={T|T < 7334419828883435520}]
I20260928 21:10:41.048887 8839 mvcc.cc:205] Tried to move back new op lower bound from 7334419828883435520 to 7334419828772896768. Current Snapshot: MvccSnapshot[applied={T|T < 7334419828883435520}]
I20260928 21:10:41.057386 8353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:41.057559 8353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:41.060314 8353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dd5680531254af7956a7da763807092: Sending DeleteTablet for 3 replicas of tablet 5047e2feec5a4ebd8db74e6d11eead30
I20260928 21:10:41.061264 8741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5047e2feec5a4ebd8db74e6d11eead30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:41 UTC) from {username='slave'} at 127.0.0.1:51556
I20260928 21:10:41.061507 8609 tablet_service.cc:1514] Processing DeleteTablet for tablet 5047e2feec5a4ebd8db74e6d11eead30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:41 UTC) from {username='slave'} at 127.0.0.1:47422
I20260928 21:10:41.062088 8848 tablet_replica.cc:330] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: stopping tablet replica
I20260928 21:10:41.062083 8477 tablet_service.cc:1514] Processing DeleteTablet for tablet 5047e2feec5a4ebd8db74e6d11eead30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:41 UTC) from {username='slave'} at 127.0.0.1:58766
I20260928 21:10:41.062500 8848 raft_consensus.cc:2229] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:41.062791 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8415
I20260928 21:10:41.062871 8848 raft_consensus.cc:2258] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:41.065820 8848 ts_tablet_manager.cc:1905] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:41.068987 8848 ts_tablet_manager.cc:1918] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:41.069176 8848 log.cc:1196] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/5047e2feec5a4ebd8db74e6d11eead30
I20260928 21:10:41.069566 8848 ts_tablet_manager.cc:1939] T 5047e2feec5a4ebd8db74e6d11eead30 P c028efc2e54d4d4195615316786b660e: Deleting consensus metadata
I20260928 21:10:41.070995 8341 catalog_manager.cc:4841] TS c028efc2e54d4d4195615316786b660e (127.21.140.66:41455): tablet 5047e2feec5a4ebd8db74e6d11eead30 (table test-table [id=356fb67e41c94df5a8658c91206a34a2]) successfully deleted
I20260928 21:10:41.071564 8847 tablet_replica.cc:330] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: stopping tablet replica
I20260928 21:10:41.071887 8847 raft_consensus.cc:2229] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:10:41.072263 8847 raft_consensus.cc:2258] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:10:41.072744 8340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:44171 (error 108)
I20260928 21:10:41.073531 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8547
I20260928 21:10:41.073791 8821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:41.073478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:44171 (local address 127.0.0.1:58772)
0928 21:10:41.073644 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:41.073707 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 21:10:41.075158 8340 catalog_manager.cc:4584] TS 6127163e3d5743eaa63ee850a4d21b00 (127.21.140.65:44171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5047e2feec5a4ebd8db74e6d11eead30: Network error: Client connection negotiation failed: client connection to 127.21.140.65:44171: connect: Connection refused (error 111)
I20260928 21:10:41.076862 8847 ts_tablet_manager.cc:1905] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:41.080876 8847 ts_tablet_manager.cc:1918] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:41.081001 8847 log.cc:1196] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/5047e2feec5a4ebd8db74e6d11eead30
I20260928 21:10:41.081419 8847 ts_tablet_manager.cc:1939] T 5047e2feec5a4ebd8db74e6d11eead30 P e5b1b7ecb4a04e92b6a6c5cb1a611401: Deleting consensus metadata
I20260928 21:10:41.081795 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8679
I20260928 21:10:41.087265 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8323
2026-09-28T21:10:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:41 dist-test-slave-d5wc krb5kdc[8853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:41 dist-test-slave-d5wc krb5kdc[8853](info): set up 2 sockets
Sep 28 21:10:41 dist-test-slave-d5wc krb5kdc[8853](info): commencing operation
krb5kdc: starting...
W20260928 21:10:42.155531 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:10:42.285923 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:38647
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:42.398078 8866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:42.398401 8866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:42.398486 8866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:42.402640 8866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:42.402731 8866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:42.402770 8866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:42.402832 8866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:42.402871 8866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:42.408545 8866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:42.409758 8866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:42.410934 8866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:42.420475 8871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:42.420555 8866 server_base.cc:1034] running on GCE node
W20260928 21:10:42.420609 8872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:42.420511 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:42.421208 8866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:42.422418 8866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:42.423594 8866 hybrid_clock.cc:648] HybridClock initialized: now 1790629842423578 us; error 33 us; skew 500 ppm
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:42.426584 8866 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:10:42.427497 8866 webserver.cc:460] Webserver started at http://127.21.140.126:45697/ using document root <none> and password file <none>
I20260928 21:10:42.428076 8866 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:42.428169 8866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:42.428400 8866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:42.430539 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "e25657a7613f4ec19f881803e470b214"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "56361c91f7cebcf2828fa227a059e662"
server_key_iv: "870059913f9b38593b137739db698157"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.431104 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "e25657a7613f4ec19f881803e470b214"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "56361c91f7cebcf2828fa227a059e662"
server_key_iv: "870059913f9b38593b137739db698157"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.434948 8866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:10:42.437451 8881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.438758 8866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:42.438903 8866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "e25657a7613f4ec19f881803e470b214"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "56361c91f7cebcf2828fa227a059e662"
server_key_iv: "870059913f9b38593b137739db698157"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.439024 8866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:42.452600 8866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:42.456489 8866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:42.456729 8866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:42.464315 8866 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:38647
I20260928 21:10:42.464362 8933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:38647 every 8 connection(s)
I20260928 21:10:42.465322 8866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:42.468127 8934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:42.472682 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8866
I20260928 21:10:42.472841 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:42.473160 22065 external_mini_cluster.cc:1428] Setting key 7c1c36bbdde496d8a8a5880d8a73cc48
I20260928 21:10:42.474123 8934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214: Bootstrap starting.
I20260928 21:10:42.476878 8934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:42.477696 8934 log.cc:824] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:42.479784 8934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214: No bootstrap required, opened a new log
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:42.483958 8934 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25657a7613f4ec19f881803e470b214" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 38647 } }
I20260928 21:10:42.484251 8934 raft_consensus.cc:374] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:42.484320 8934 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25657a7613f4ec19f881803e470b214, State: Initialized, Role: FOLLOWER
I20260928 21:10:42.484867 8934 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25657a7613f4ec19f881803e470b214" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 38647 } }
I20260928 21:10:42.485033 8934 raft_consensus.cc:388] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:42.485103 8934 raft_consensus.cc:482] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:42.485199 8934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:42.486392 8934 raft_consensus.cc:504] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25657a7613f4ec19f881803e470b214" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 38647 } }
I20260928 21:10:42.486778 8934 leader_election.cc:302] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e25657a7613f4ec19f881803e470b214; no voters:
I20260928 21:10:42.487143 8934 leader_election.cc:288] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:42.487306 8939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:42.487537 8939 raft_consensus.cc:686] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [term 1 LEADER]: Becoming Leader. State: Replica: e25657a7613f4ec19f881803e470b214, State: Running, Role: LEADER
I20260928 21:10:42.487938 8939 consensus_queue.cc:237] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25657a7613f4ec19f881803e470b214" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 38647 } }
I20260928 21:10:42.488407 8934 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:42.489259 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:42.474597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.21.140.126:38647)
0928 21:10:42.475029 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:42.475051 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:42.475102 (+ 51us) server_negotiation.cc:407] Connection header received
0928 21:10:42.475947 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:42.475970 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:42.476395 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:42.476761 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:42.477662 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.479037 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.479947 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.480624 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.480782 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:42.483642 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:42.483648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:42.483665 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:42.483697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:42.485732 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:42.486224 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:42.486233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:42.486242 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:42.486391 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:42.486753 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:42.486757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:42.486758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:42.487354 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:42.487547 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:42.487942 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 21:10:42.488259 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260928 21:10:42.489274 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e25657a7613f4ec19f881803e470b214. Latest consensus state: current_term: 1 leader_uuid: "e25657a7613f4ec19f881803e470b214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25657a7613f4ec19f881803e470b214" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 38647 } } }
I20260928 21:10:42.489394 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:42.489746 8948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:42.489696 8940 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e25657a7613f4ec19f881803e470b214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25657a7613f4ec19f881803e470b214" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 38647 } } }
I20260928 21:10:42.489806 8940 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:42.493018 8948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:42.499051 8948 catalog_manager.cc:1347] Generated new cluster ID: a990796a46fc4023a3ed011c1a4c208e
I20260928 21:10:42.499121 8948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:42.507212 8948 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:42.508405 8948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:42.530367 8948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214: Generated new TSK 0
I20260928 21:10:42.531277 8948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.140.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
Entry for principal kudu/127.21.140.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab.
I20260928 21:10:42.565461 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:38647
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:42.678107 8960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:42.678398 8960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:42.678476 8960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:42.682593 8960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:42.682691 8960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:42.682785 8960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:42.688139 8960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.65.keytab
--principal=kudu/127.21.140.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:42.689327 8960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:42.690483 8960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:42.700073 8966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:42.700081 8968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:42.700119 8965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:42.700238 8960 server_base.cc:1034] running on GCE node
I20260928 21:10:42.700760 8960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:42.701356 8960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:42.702543 8960 hybrid_clock.cc:648] HybridClock initialized: now 1790629842702523 us; error 36 us; skew 500 ppm
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:42.705372 8960 init.cc:377] Logged in from keytab as kudu/127.21.140.65@KRBTEST.COM (short username kudu)
I20260928 21:10:42.706359 8960 webserver.cc:460] Webserver started at http://127.21.140.65:46005/ using document root <none> and password file <none>
I20260928 21:10:42.706931 8960 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:42.707022 8960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:42.707260 8960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:42.709524 8960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "11d116f10776477da193a0d61226d51b"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "ff5a598ff35a125bb49273a81bb63b90"
server_key_iv: "d159805ae8858de09e8f1651d5c1fae6"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.710157 8960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "11d116f10776477da193a0d61226d51b"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "ff5a598ff35a125bb49273a81bb63b90"
server_key_iv: "d159805ae8858de09e8f1651d5c1fae6"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.713857 8960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:42.716341 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.717541 8960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:10:42.717715 8960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "11d116f10776477da193a0d61226d51b"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "ff5a598ff35a125bb49273a81bb63b90"
server_key_iv: "d159805ae8858de09e8f1651d5c1fae6"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.717828 8960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:42.742543 8960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:42.746097 8960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:42.746315 8960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:42.746937 8960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:42.747807 8960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:42.747880 8960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.747951 8960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:42.747999 8960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.757213 8960 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:36191
I20260928 21:10:42.757236 9088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:36191 every 8 connection(s)
I20260928 21:10:42.758155 8960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:42.762684 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 8960
I20260928 21:10:42.762835 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:10:42.763139 22065 external_mini_cluster.cc:1428] Setting key d57073a5d97038719eb85982319c11ba
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.65@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.66@KRBTEST.COM; defaulting to no policy
I20260928 21:10:42.772368 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:42.759987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:38349 (local address 127.21.140.126:38647)
0928 21:10:42.760139 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:42.760143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:42.761045 (+ 902us) server_negotiation.cc:407] Connection header received
0928 21:10:42.762512 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:42.762515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:42.762573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:42.762701 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:42.764061 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.764690 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.765848 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.766149 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.766261 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:42.768761 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:42.768765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:42.768770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:42.768799 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:42.770293 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:42.770808 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:42.770812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:42.770814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:42.770899 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:42.771252 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:42.771257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:42.771259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:42.771442 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:42.771552 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:42.772075 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 21:10:42.772203 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:10:42.773268 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:42.760359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:38647 (local address 127.21.140.65:38349)
0928 21:10:42.760891 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:42.760935 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:42.762294 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:42.762881 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:42.762889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:42.763336 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:42.763915 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:42.763929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.764820 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.764828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:42.765718 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:42.765727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.766302 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.766306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:42.766477 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:42.767127 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:42.767143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:42.768587 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:42.770416 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:42.770422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:42.770438 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:42.770701 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:42.770994 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:42.770999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:42.771001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:42.771152 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:42.771550 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:42.771557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:42.771810 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 21:10:42.772033 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 21:10:42.772380 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":177,"threads_started":1}
I20260928 21:10:42.774547 9089 heartbeater.cc:347] Connected to a master server at 127.21.140.126:38647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.66@KRBTEST.COM" created.
I20260928 21:10:42.774837 9089 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:42.775470 9089 heartbeater.cc:511] Master 127.21.140.126:38647 requested a full tablet report, sending...
I20260928 21:10:42.777071 8898 ts_manager.cc:194] Registered new tserver with Master: 11d116f10776477da193a0d61226d51b (127.21.140.65:36191)
I20260928 21:10:42.778270 8898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.65@KRBTEST.COM'} at 127.21.140.65:38349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
Entry for principal kudu/127.21.140.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab.
I20260928 21:10:42.793308 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:38647
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:42.906970 9094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:42.907253 9094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:42.907299 9094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:42.911568 9094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:42.911634 9094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:42.911739 9094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:42.917191 9094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.66.keytab
--principal=kudu/127.21.140.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:42.918360 9094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:42.919389 9094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:42.930127 9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:42.930225 9094 server_base.cc:1034] running on GCE node
W20260928 21:10:42.930140 9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:42.930151 9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:42.930862 9094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:42.931555 9094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:42.932716 9094 hybrid_clock.cc:648] HybridClock initialized: now 1790629842932701 us; error 26 us; skew 500 ppm
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:42.935616 9094 init.cc:377] Logged in from keytab as kudu/127.21.140.66@KRBTEST.COM (short username kudu)
I20260928 21:10:42.936530 9094 webserver.cc:460] Webserver started at http://127.21.140.66:38761/ using document root <none> and password file <none>
I20260928 21:10:42.937069 9094 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:42.937136 9094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:42.937299 9094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:42.939402 9094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "0fc9a4baaae64881a4b9b29af4b88176"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "7a4dc217a065db7b6fb9d68bdcf0b9cb"
server_key_iv: "5c3c2db0118dd7950f9fc940e8d12754"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.939916 9094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "0fc9a4baaae64881a4b9b29af4b88176"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "7a4dc217a065db7b6fb9d68bdcf0b9cb"
server_key_iv: "5c3c2db0118dd7950f9fc940e8d12754"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.943661 9094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:42.946061 9109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.947382 9094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:42.947618 9094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "0fc9a4baaae64881a4b9b29af4b88176"
format_stamp: "Formatted at 2026-09-28 21:10:42 on dist-test-slave-d5wc"
server_key: "7a4dc217a065db7b6fb9d68bdcf0b9cb"
server_key_iv: "5c3c2db0118dd7950f9fc940e8d12754"
server_key_version: "encryptionkey@0"
I20260928 21:10:42.947739 9094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:42.961206 9094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:42.964184 9094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:42.964393 9094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:42.964983 9094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:42.965808 9094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:42.965879 9094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.965950 9094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:42.966015 9094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:42.975509 9094 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:45061
I20260928 21:10:42.975538 9222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:45061 every 8 connection(s)
I20260928 21:10:42.976424 9094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:42.979681 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9094
I20260928 21:10:42.979838 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:10:42.980130 22065 external_mini_cluster.cc:1428] Setting key 5067e83d8a4ff1514593fca1f6da93e1
Sep 28 21:10:42 dist-test-slave-d5wc krb5kdc[8853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.66@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.140.67@KRBTEST.COM; defaulting to no policy
I20260928 21:10:42.992240 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:42.978191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:39251 (local address 127.21.140.126:38647)
0928 21:10:42.978339 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:42.978343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:42.979292 (+ 949us) server_negotiation.cc:407] Connection header received
0928 21:10:42.980946 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:42.980949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:42.980991 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:42.981054 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:42.982470 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.983333 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.985025 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.985445 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.985571 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:42.988053 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:42.988060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:42.988064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:42.988107 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:42.989830 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:42.990442 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:42.990448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:42.990452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:42.990555 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:42.990985 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:42.990989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:42.990991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:42.991217 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:42.991359 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:42.991944 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 21:10:42.992075 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:10:42.993512 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:42.978698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:38647 (local address 127.21.140.66:39251)
0928 21:10:42.979135 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:42.979179 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:42.980721 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:42.981258 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:42.981267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:42.981760 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:42.982312 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:42.982326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.983503 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.983509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:42.984885 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:42.984896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:42.985600 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:42.985603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:42.985745 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:42.986437 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:42.986453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:42.987866 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:42.989999 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:42.990008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:42.990027 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:42.990313 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:42.990659 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:42.990662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:42.990666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:42.990834 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:42.991358 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:42.991365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:42.991632 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 21:10:42.991883 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 21:10:42.992241 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.67@KRBTEST.COM" created.
I20260928 21:10:42.994838 9223 heartbeater.cc:347] Connected to a master server at 127.21.140.126:38647
I20260928 21:10:42.995095 9223 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:42.995712 9223 heartbeater.cc:511] Master 127.21.140.126:38647 requested a full tablet report, sending...
I20260928 21:10:42.996905 8898 ts_manager.cc:194] Registered new tserver with Master: 0fc9a4baaae64881a4b9b29af4b88176 (127.21.140.66:45061)
I20260928 21:10:42.997780 8898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.66@KRBTEST.COM'} at 127.21.140.66:39251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
Entry for principal kudu/127.21.140.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab.
I20260928 21:10:43.011530 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:38647
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:43.123579 9228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:43.123844 9228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:43.123888 9228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:43.127921 9228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:43.127980 9228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:43.128039 9228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:43.133248 9228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.67.keytab
--principal=kudu/127.21.140.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:43.134317 9228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:43.135274 9228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:43.145427 9234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:43.145427 9236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:43.145496 9233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:43.145768 9228 server_base.cc:1034] running on GCE node
I20260928 21:10:43.146250 9228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:43.146916 9228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:43.148078 9228 hybrid_clock.cc:648] HybridClock initialized: now 1790629843148043 us; error 48 us; skew 500 ppm
Sep 28 21:10:43 dist-test-slave-d5wc krb5kdc[8853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:43.150962 9228 init.cc:377] Logged in from keytab as kudu/127.21.140.67@KRBTEST.COM (short username kudu)
I20260928 21:10:43.151932 9228 webserver.cc:460] Webserver started at http://127.21.140.67:35581/ using document root <none> and password file <none>
I20260928 21:10:43.152489 9228 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:43.152590 9228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:43.152817 9228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:43.154909 9228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "dcb007f25d78423abd8ccb95f728fb85"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "3eaa00436745c22ff5ec070029b27937"
server_key_iv: "34c3f3c3fe72cd42c3afa289f0d8b1fc"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.155471 9228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "dcb007f25d78423abd8ccb95f728fb85"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "3eaa00436745c22ff5ec070029b27937"
server_key_iv: "34c3f3c3fe72cd42c3afa289f0d8b1fc"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.159178 9228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 21:10:43.161648 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.162925 9228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:10:43.163065 9228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "dcb007f25d78423abd8ccb95f728fb85"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "3eaa00436745c22ff5ec070029b27937"
server_key_iv: "34c3f3c3fe72cd42c3afa289f0d8b1fc"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.163187 9228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:43.182463 9228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:43.185869 9228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:43.186112 9228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:43.186731 9228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:43.187580 9228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:43.187659 9228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.187744 9228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:43.187794 9228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.196847 9228 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:37959
I20260928 21:10:43.196877 9356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:37959 every 8 connection(s)
I20260928 21:10:43.197857 9228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:10:43 dist-test-slave-d5wc krb5kdc[8853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.67@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:10:43.208106 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9228
I20260928 21:10:43.208269 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:10:43.208657 22065 external_mini_cluster.cc:1428] Setting key 14802a694d6fe805dfc62d2a0398531d
I20260928 21:10:43.212013 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:43419 (local address 127.21.140.126:38647)
0928 21:10:43.199889 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.199894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.200835 (+ 941us) server_negotiation.cc:407] Connection header received
0928 21:10:43.202142 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.202145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.202194 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.202269 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:43.203456 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.203975 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.205089 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.205290 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.205367 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.207932 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:43.207936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:43.207939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:43.207971 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:43.209677 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:43.210339 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:43.210341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:43.210343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:43.210420 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:43.210870 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:43.210876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:43.210880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:43.211154 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:43.211236 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.211789 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.211895 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:10:43.212874 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.200160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:38647 (local address 127.21.140.67:43419)
0928 21:10:43.200673 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.200717 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.201919 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.202427 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.202434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.202775 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:43.203304 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.203326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.204080 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.204082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.204970 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.204978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.205419 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.205421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.205561 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.206177 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:10:43.206194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:43.207713 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:43.209799 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:43.209805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:43.209818 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:43.210229 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:43.210509 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:10:43.210512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:10:43.210514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:10:43.210736 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:10:43.211253 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:43.211258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:43.211515 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.211725 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.212007 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260928 21:10:43.214120 9357 heartbeater.cc:347] Connected to a master server at 127.21.140.126:38647
I20260928 21:10:43.214372 9357 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:43.214871 9357 heartbeater.cc:511] Master 127.21.140.126:38647 requested a full tablet report, sending...
I20260928 21:10:43.215911 8898 ts_manager.cc:194] Registered new tserver with Master: dcb007f25d78423abd8ccb95f728fb85 (127.21.140.67:37959)
I20260928 21:10:43.216504 8898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.140.67@KRBTEST.COM'} at 127.21.140.67:43419
I20260928 21:10:43.222831 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:10:43.232446 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.224785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.21.140.126:38647)
0928 21:10:43.224930 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.224936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.225110 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:10:43.225261 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.225265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.225308 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.225403 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:43.226161 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.226680 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.227645 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.227885 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.227994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.229367 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:43.229371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:43.229374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:43.229406 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:43.230755 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:43.231176 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:43.231180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:43.231182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:43.231262 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:43.231547 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:43.231550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:43.231552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:43.231773 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:43.231917 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.232165 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.232279 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:10:43.235334 8898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:10:43.238843 8898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:10:43.253633 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.248278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36191 (local address 127.0.0.1:52672)
0928 21:10:43.248843 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.248857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.249006 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.249663 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.249665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.249678 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:43.249748 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.249752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.250816 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.250823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.252238 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.252255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.253067 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.253070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.253189 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.253227 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.253338 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.253459 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":101,"threads_started":1}
I20260928 21:10:43.253785 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.247665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.21.140.67:37959)
0928 21:10:43.248771 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.248785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.248807 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:10:43.248884 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.248891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.249100 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.249223 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:43.249513 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.250709 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.252402 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.253241 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.253347 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.253464 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.253529 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.253637 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":169,"threads_started":1}
I20260928 21:10:43.254155 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.248704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.21.140.65:36191)
0928 21:10:43.249109 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.249128 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.249145 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:10:43.249234 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.249238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.249402 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.249541 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:43.249892 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.250686 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.252363 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.252940 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.253068 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.253238 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.253918 (+ 680us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.254040 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":90,"threads_started":1}
I20260928 21:10:43.253665 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.247444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:37959 (local address 127.0.0.1:55744)
0928 21:10:43.248416 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.248469 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.248615 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.249246 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.249252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.249274 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:43.249408 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.249417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.250861 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.250864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.252286 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.252296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.253348 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.253351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.253428 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.253447 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.253505 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.253562 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":251,"thread_start_us":95,"threads_started":1}
I20260928 21:10:43.257519 9023 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4e5ca25b82482892d8471541649f2f (DEFAULT_TABLE table=test-table [id=7e52fb621f2547e4a453f50cf3f25940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:43.257519 9291 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4e5ca25b82482892d8471541649f2f (DEFAULT_TABLE table=test-table [id=7e52fb621f2547e4a453f50cf3f25940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:43.258420 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.247750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.21.140.66:45061)
0928 21:10:43.248160 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.248169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.248562 (+ 393us) server_negotiation.cc:407] Connection header received
0928 21:10:43.249513 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.249518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.249690 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.249840 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:43.250056 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.251157 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.253687 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.254517 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.254631 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.254759 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.258126 (+ 3367us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.258248 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":107,"threads_started":1}
I20260928 21:10:43.258617 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.247444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:45061 (local address 127.0.0.1:59434)
0928 21:10:43.248418 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.248469 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.249350 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.249836 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.249839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.249857 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:43.249936 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.249942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.252431 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.252433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.253550 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.253560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.254822 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.254825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.254912 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.254933 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.255015 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.255079 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":101,"threads_started":1}
I20260928 21:10:43.258718 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4e5ca25b82482892d8471541649f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:43.258718 9291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4e5ca25b82482892d8471541649f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:43.261221 9157 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4e5ca25b82482892d8471541649f2f (DEFAULT_TABLE table=test-table [id=7e52fb621f2547e4a453f50cf3f25940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:43.262240 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4e5ca25b82482892d8471541649f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:43.266634 9375 tablet_bootstrap.cc:492] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: Bootstrap starting.
I20260928 21:10:43.267666 9374 tablet_bootstrap.cc:492] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Bootstrap starting.
I20260928 21:10:43.269148 9375 tablet_bootstrap.cc:654] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:43.269459 9374 tablet_bootstrap.cc:654] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:43.269737 9376 tablet_bootstrap.cc:492] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Bootstrap starting.
I20260928 21:10:43.270856 9374 log.cc:824] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:43.271260 9375 log.cc:824] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:43.271986 9376 tablet_bootstrap.cc:654] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:43.272974 9375 tablet_bootstrap.cc:492] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: No bootstrap required, opened a new log
I20260928 21:10:43.273075 9374 tablet_bootstrap.cc:492] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: No bootstrap required, opened a new log
I20260928 21:10:43.273233 9375 ts_tablet_manager.cc:1397] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:10:43.273265 9374 ts_tablet_manager.cc:1397] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:10:43.273288 9376 log.cc:824] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:43.275451 9376 tablet_bootstrap.cc:492] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: No bootstrap required, opened a new log
I20260928 21:10:43.275652 9376 ts_tablet_manager.cc:1397] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:10:43.276542 9375 raft_consensus.cc:348] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.276822 9375 raft_consensus.cc:374] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:43.276882 9375 raft_consensus.cc:729] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d116f10776477da193a0d61226d51b, State: Initialized, Role: FOLLOWER
I20260928 21:10:43.277088 9374 raft_consensus.cc:348] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.277348 9374 raft_consensus.cc:374] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:43.277400 9374 raft_consensus.cc:729] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb007f25d78423abd8ccb95f728fb85, State: Initialized, Role: FOLLOWER
I20260928 21:10:43.277391 9375 consensus_queue.cc:260] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.277827 9374 consensus_queue.cc:260] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.278204 9089 heartbeater.cc:503] Master 127.21.140.126:38647 was elected leader, sending a full tablet report...
I20260928 21:10:43.278535 9357 heartbeater.cc:503] Master 127.21.140.126:38647 was elected leader, sending a full tablet report...
I20260928 21:10:43.278568 9375 ts_tablet_manager.cc:1428] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:10:43.279623 9374 ts_tablet_manager.cc:1428] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:10:43.280583 9376 raft_consensus.cc:348] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.280784 9376 raft_consensus.cc:374] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:43.280859 9376 raft_consensus.cc:729] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc9a4baaae64881a4b9b29af4b88176, State: Initialized, Role: FOLLOWER
I20260928 21:10:43.281324 9376 consensus_queue.cc:260] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.282371 9223 heartbeater.cc:503] Master 127.21.140.126:38647 was elected leader, sending a full tablet report...
I20260928 21:10:43.283387 9376 ts_tablet_manager.cc:1428] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 21:10:43.310585 9382 raft_consensus.cc:482] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:43.310822 9382 raft_consensus.cc:504] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.311944 9382 leader_election.cc:288] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb007f25d78423abd8ccb95f728fb85 (127.21.140.67:37959), 11d116f10776477da193a0d61226d51b (127.21.140.65:36191)
I20260928 21:10:43.316912 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.312342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:36193 (local address 127.21.140.67:37959)
0928 21:10:43.312492 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.312499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.312801 (+ 302us) server_negotiation.cc:407] Connection header received
0928 21:10:43.312894 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.312901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.312943 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.313003 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:43.313293 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.314073 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.315644 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.316251 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.316377 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.316430 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.316705 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.316775 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:10:43.316912 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.312339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:60449 (local address 127.21.140.65:36191)
0928 21:10:43.312461 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.312466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.312480 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:10:43.312639 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.312642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.312690 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.312758 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:43.313136 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.313939 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.315644 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.316252 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.316377 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.316430 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.316705 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.316775 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:10:43.317021 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.312214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36191 (local address 127.21.140.66:60449)
0928 21:10:43.312371 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.312388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.312488 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.312795 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.312799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.312820 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:43.312965 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.312971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.314059 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.314062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.315497 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.315509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.316377 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.316381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.316531 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.316592 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.316670 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.316835 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":165,"spinlock_wait_cycles":768}
I20260928 21:10:43.317030 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.312214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:37959 (local address 127.21.140.66:36193)
0928 21:10:43.312687 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.312698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.312811 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.313044 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.313049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.313072 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:43.313187 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.313192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.314175 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.314178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.315497 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.315509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.316377 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.316381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.316506 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.316591 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.316670 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.316835 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":254,"thread_start_us":125,"threads_started":1}
I20260928 21:10:43.317744 9311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4e5ca25b82482892d8471541649f2f" candidate_uuid: "0fc9a4baaae64881a4b9b29af4b88176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb007f25d78423abd8ccb95f728fb85" is_pre_election: true
I20260928 21:10:43.317744 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4e5ca25b82482892d8471541649f2f" candidate_uuid: "0fc9a4baaae64881a4b9b29af4b88176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d116f10776477da193a0d61226d51b" is_pre_election: true
I20260928 21:10:43.318086 9043 raft_consensus.cc:2454] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fc9a4baaae64881a4b9b29af4b88176 in term 0.
I20260928 21:10:43.318094 9311 raft_consensus.cc:2454] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fc9a4baaae64881a4b9b29af4b88176 in term 0.
I20260928 21:10:43.318640 9111 leader_election.cc:302] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0fc9a4baaae64881a4b9b29af4b88176, 11d116f10776477da193a0d61226d51b, dcb007f25d78423abd8ccb95f728fb85; no voters:
I20260928 21:10:43.318950 9382 raft_consensus.cc:2781] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:10:43.319029 9382 raft_consensus.cc:482] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:43.319060 9382 raft_consensus.cc:3037] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:43.320134 9382 raft_consensus.cc:504] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.320570 9382 leader_election.cc:288] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [CANDIDATE]: Term 1 election: Requested vote from peers dcb007f25d78423abd8ccb95f728fb85 (127.21.140.67:37959), 11d116f10776477da193a0d61226d51b (127.21.140.65:36191)
I20260928 21:10:43.320926 9311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4e5ca25b82482892d8471541649f2f" candidate_uuid: "0fc9a4baaae64881a4b9b29af4b88176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb007f25d78423abd8ccb95f728fb85"
I20260928 21:10:43.320966 9043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4e5ca25b82482892d8471541649f2f" candidate_uuid: "0fc9a4baaae64881a4b9b29af4b88176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d116f10776477da193a0d61226d51b"
I20260928 21:10:43.321036 9311 raft_consensus.cc:3037] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:43.321123 9043 raft_consensus.cc:3037] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:43.322352 9043 raft_consensus.cc:2454] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fc9a4baaae64881a4b9b29af4b88176 in term 1.
I20260928 21:10:43.322363 9311 raft_consensus.cc:2454] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fc9a4baaae64881a4b9b29af4b88176 in term 1.
I20260928 21:10:43.322672 9111 leader_election.cc:302] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fc9a4baaae64881a4b9b29af4b88176, 11d116f10776477da193a0d61226d51b; no voters:
I20260928 21:10:43.322899 9382 raft_consensus.cc:2781] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:43.323196 9382 raft_consensus.cc:686] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 1 LEADER]: Becoming Leader. State: Replica: 0fc9a4baaae64881a4b9b29af4b88176, State: Running, Role: LEADER
I20260928 21:10:43.323621 9382 consensus_queue.cc:237] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } }
I20260928 21:10:43.327102 8897 catalog_manager.cc:5495] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fc9a4baaae64881a4b9b29af4b88176 (127.21.140.66). New cstate: current_term: 1 leader_uuid: "0fc9a4baaae64881a4b9b29af4b88176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc9a4baaae64881a4b9b29af4b88176" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 45061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 } health_report { overall_health: UNKNOWN } } }
I20260928 21:10:43.352095 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.348355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.21.140.66:45061)
0928 21:10:43.348496 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.348499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.348514 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:10:43.348620 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.348623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.348682 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.348766 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:10:43.348988 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.349491 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.350397 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.350617 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.350700 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.351147 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:10:43.351647 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:10:43.351808 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.351866 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.351932 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:10:43.358376 9043 raft_consensus.cc:1261] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 1 FOLLOWER]: Refusing update from remote peer 0fc9a4baaae64881a4b9b29af4b88176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:43.358376 9311 raft_consensus.cc:1261] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 1 FOLLOWER]: Refusing update from remote peer 0fc9a4baaae64881a4b9b29af4b88176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:43.359154 9382 consensus_queue.cc:1035] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb007f25d78423abd8ccb95f728fb85" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:43.359378 9384 consensus_queue.cc:1035] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d116f10776477da193a0d61226d51b" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:43.367965 9389 mvcc.cc:205] Tried to move back new op lower bound from 7334419838388187136 to 7334419838256865280. Current Snapshot: MvccSnapshot[applied={T|T < 7334419838388187136}]
I20260928 21:10:43.368237 9388 mvcc.cc:205] Tried to move back new op lower bound from 7334419838388187136 to 7334419838256865280. Current Snapshot: MvccSnapshot[applied={T|T < 7334419838388187136}]
I20260928 21:10:43.376420 8896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:43.376629 8896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:43.381412 8896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e25657a7613f4ec19f881803e470b214: Sending DeleteTablet for 3 replicas of tablet 9b4e5ca25b82482892d8471541649f2f
I20260928 21:10:43.382504 9291 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4e5ca25b82482892d8471541649f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:43 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:55744
I20260928 21:10:43.383234 9397 tablet_replica.cc:330] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: stopping tablet replica
I20260928 21:10:43.383028 9023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4e5ca25b82482892d8471541649f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:43 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:52672
I20260928 21:10:43.383621 9397 raft_consensus.cc:2229] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:43.384004 9397 raft_consensus.cc:2258] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:43.384207 9398 tablet_replica.cc:330] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b: stopping tablet replica
I20260928 21:10:43.384779 9398 raft_consensus.cc:2229] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:43.385054 9157 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4e5ca25b82482892d8471541649f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:43 UTC) from {username='kudu', principal='kudu/127.21.140.126@KRBTEST.COM'} at 127.0.0.1:59434
I20260928 21:10:43.385457 9398 raft_consensus.cc:2258] T 9b4e5ca25b82482892d8471541649f2f P 11d116f10776477da193a0d61226d51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:43.386672 9399 tablet_replica.cc:330] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: stopping tablet replica
I20260928 21:10:43.386966 9399 raft_consensus.cc:2229] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:10:43.387271 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8960
I20260928 21:10:43.387271 9397 ts_tablet_manager.cc:1905] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:43.387346 9399 raft_consensus.cc:2258] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:43.389304 9399 ts_tablet_manager.cc:1905] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:43.391321 9397 ts_tablet_manager.cc:1918] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:43.391424 9397 log.cc:1196] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/9b4e5ca25b82482892d8471541649f2f
I20260928 21:10:43.391810 9397 ts_tablet_manager.cc:1939] T 9b4e5ca25b82482892d8471541649f2f P dcb007f25d78423abd8ccb95f728fb85: Deleting consensus metadata
I20260928 21:10:43.392098 9399 ts_tablet_manager.cc:1918] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:43.392199 9399 log.cc:1196] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/9b4e5ca25b82482892d8471541649f2f
I20260928 21:10:43.393241 8885 catalog_manager.cc:4841] TS dcb007f25d78423abd8ccb95f728fb85 (127.21.140.67:37959): tablet 9b4e5ca25b82482892d8471541649f2f (table test-table [id=7e52fb621f2547e4a453f50cf3f25940]) successfully deleted
I20260928 21:10:43.393702 9399 ts_tablet_manager.cc:1939] T 9b4e5ca25b82482892d8471541649f2f P 0fc9a4baaae64881a4b9b29af4b88176: Deleting consensus metadata
I20260928 21:10:43.394903 8883 catalog_manager.cc:4841] TS 0fc9a4baaae64881a4b9b29af4b88176 (127.21.140.66:45061): tablet 9b4e5ca25b82482892d8471541649f2f (table test-table [id=7e52fb621f2547e4a453f50cf3f25940]) successfully deleted
W20260928 21:10:43.397291 8883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:36191 (error 108)
I20260928 21:10:43.397466 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9094
I20260928 21:10:43.398929 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.398532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:36191 (local address 127.0.0.1:52682)
0928 21:10:43.398748 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.398824 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:36191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260928 21:10:43.399101 8883 catalog_manager.cc:4584] TS 11d116f10776477da193a0d61226d51b (127.21.140.65:36191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b4e5ca25b82482892d8471541649f2f: Network error: Client connection negotiation failed: client connection to 127.21.140.65:36191: connect: Connection refused (error 111)
I20260928 21:10:43.403724 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9228
I20260928 21:10:43.409171 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 8866
2026-09-28T21:10:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T21:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:43Z Disabled control of system clock
I20260928 21:10:43.440425 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33965
--webserver_interface=127.21.140.126
--webserver_port=0
--builtin_ntp_servers=127.21.140.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:33965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:10:43.553376 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:43.553671 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:43.553709 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:43.557670 9404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:43.557729 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:43.557745 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:43.557761 9404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:43.557776 9404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:43.562569 9404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:33965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:43.563696 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:43.565037 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:43.573628 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:43.573685 9404 server_base.cc:1034] running on GCE node
W20260928 21:10:43.573637 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:43.573654 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:43.574602 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:43.575883 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:43.577124 9404 hybrid_clock.cc:648] HybridClock initialized: now 1790629843577097 us; error 49 us; skew 500 ppm
I20260928 21:10:43.579018 9404 webserver.cc:460] Webserver started at http://127.21.140.126:35715/ using document root <none> and password file <none>
I20260928 21:10:43.579596 9404 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:43.579690 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:43.579921 9404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:43.582022 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "cec49aba05e64250a4f2209daf7047fb"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "62489ab68e1fc7dd90beaeb799892a62"
server_key_iv: "f182493f404f182946758d94129894ee"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.582669 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "cec49aba05e64250a4f2209daf7047fb"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "62489ab68e1fc7dd90beaeb799892a62"
server_key_iv: "f182493f404f182946758d94129894ee"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.586419 9404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 21:10:43.588768 9418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.590000 9404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:43.590154 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "cec49aba05e64250a4f2209daf7047fb"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "62489ab68e1fc7dd90beaeb799892a62"
server_key_iv: "f182493f404f182946758d94129894ee"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.590270 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:43.601743 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:43.602494 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:43.602705 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:43.610519 9404 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:33965
I20260928 21:10:43.610536 9470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:33965 every 8 connection(s)
I20260928 21:10:43.611476 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:43.614588 9471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:43.616878 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9404
I20260928 21:10:43.617017 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:43.617318 22065 external_mini_cluster.cc:1428] Setting key 4862b09ca435edf7ba94849db3a30048
I20260928 21:10:43.620935 9471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb: Bootstrap starting.
I20260928 21:10:43.623588 9471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:43.624517 9471 log.cc:824] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:43.626801 9471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb: No bootstrap required, opened a new log
I20260928 21:10:43.626972 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.618828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.21.140.126:33965)
0928 21:10:43.619421 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.619436 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.619470 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:10:43.620124 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.620142 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.620546 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.620833 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:43.621239 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.622276 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.623429 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.623966 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.624088 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.625052 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:43.625059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:43.625075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:43.625577 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:43.625655 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:43.625676 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:43.625753 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.625916 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.626196 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":128,"threads_started":1}
I20260928 21:10:43.630455 9471 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec49aba05e64250a4f2209daf7047fb" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33965 } }
I20260928 21:10:43.630679 9471 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:43.630713 9471 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec49aba05e64250a4f2209daf7047fb, State: Initialized, Role: FOLLOWER
I20260928 21:10:43.631155 9471 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec49aba05e64250a4f2209daf7047fb" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33965 } }
I20260928 21:10:43.631342 9471 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:43.631412 9471 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:43.631524 9471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:43.632515 9471 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec49aba05e64250a4f2209daf7047fb" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33965 } }
I20260928 21:10:43.632887 9471 leader_election.cc:302] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec49aba05e64250a4f2209daf7047fb; no voters:
I20260928 21:10:43.633206 9471 leader_election.cc:288] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:43.633335 9476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:43.633546 9476 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [term 1 LEADER]: Becoming Leader. State: Replica: cec49aba05e64250a4f2209daf7047fb, State: Running, Role: LEADER
I20260928 21:10:43.633852 9476 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec49aba05e64250a4f2209daf7047fb" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33965 } }
I20260928 21:10:43.634464 9471 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:43.635551 9477 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec49aba05e64250a4f2209daf7047fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec49aba05e64250a4f2209daf7047fb" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33965 } } }
I20260928 21:10:43.635551 9478 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec49aba05e64250a4f2209daf7047fb. Latest consensus state: current_term: 1 leader_uuid: "cec49aba05e64250a4f2209daf7047fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec49aba05e64250a4f2209daf7047fb" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 33965 } } }
I20260928 21:10:43.635718 9477 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:43.635718 9478 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:43.636322 9485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:43.639477 9485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:43.644870 9485 catalog_manager.cc:1347] Generated new cluster ID: 8bbd73053e4e49e78da587a92638cb30
I20260928 21:10:43.644949 9485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:43.654663 9485 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:43.655778 9485 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:43.664558 9485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb: Generated new TSK 0
I20260928 21:10:43.665371 9485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:43.673411 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.65:0
--local_ip_for_outbound_sockets=127.21.140.65
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33965
--builtin_ntp_servers=127.21.140.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:10:43.786881 9495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:43.787145 9495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:43.787184 9495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:43.791244 9495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:43.791301 9495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:43.791383 9495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.65
I20260928 21:10:43.796594 9495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.140.65
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.65
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:43.797732 9495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:43.799258 9495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:10:43.809275 9495 server_base.cc:1034] running on GCE node
W20260928 21:10:43.809227 9501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:43.809197 9500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:43.809216 9503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:43.809913 9495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:43.810658 9495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:43.811853 9495 hybrid_clock.cc:648] HybridClock initialized: now 1790629843811820 us; error 47 us; skew 500 ppm
I20260928 21:10:43.813675 9495 webserver.cc:460] Webserver started at http://127.21.140.65:39529/ using document root <none> and password file <none>
I20260928 21:10:43.814328 9495 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:43.814428 9495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:43.814678 9495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:43.816728 9495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data/instance:
uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "4c1e9f59800dcb92100a61894b51d18a"
server_key_iv: "c1d2888e9b7a83e39a3b243fd72e25c1"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.817287 9495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance:
uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "4c1e9f59800dcb92100a61894b51d18a"
server_key_iv: "c1d2888e9b7a83e39a3b243fd72e25c1"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.821029 9495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:10:43.823446 9509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.824641 9495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:10:43.824785 9495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal
uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff"
format_stamp: "Formatted at 2026-09-28 21:10:43 on dist-test-slave-d5wc"
server_key: "4c1e9f59800dcb92100a61894b51d18a"
server_key_iv: "c1d2888e9b7a83e39a3b243fd72e25c1"
server_key_version: "encryptionkey@0"
I20260928 21:10:43.824906 9495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:43.859786 9495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:43.860644 9495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:43.860867 9495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:43.861495 9495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:43.862577 9495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:43.862660 9495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.862728 9495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:43.862779 9495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:43.872037 9495 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.65:38173
I20260928 21:10:43.872069 9622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.65:38173 every 8 connection(s)
I20260928 21:10:43.873018 9495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/data/info.pb
I20260928 21:10:43.880721 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9495
I20260928 21:10:43.880887 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-0/wal/instance
I20260928 21:10:43.881207 22065 external_mini_cluster.cc:1428] Setting key 6634b573aa27e1b83a204ba3617bfba0
I20260928 21:10:43.882984 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.875028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.65:51677 (local address 127.21.140.126:33965)
0928 21:10:43.875201 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:43.875207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:43.876440 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 21:10:43.878105 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:43.878108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:43.878157 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:43.878224 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:43.879126 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.879640 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.880821 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.881158 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.881274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.882461 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:43.882465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:43.882468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:43.882481 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:43.882510 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:43.882518 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:43.882615 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:43.882782 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 21:10:43.882844 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 21:10:43.883261 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.66:0
--local_ip_for_outbound_sockets=127.21.140.66
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33965
--builtin_ntp_servers=127.21.140.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:10:43.883962 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:43.875501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33965 (local address 127.21.140.65:51677)
0928 21:10:43.876261 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:43.876316 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:43.877810 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:43.878394 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:43.878402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:43.878492 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:43.878985 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.878997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.879746 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.879749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.880676 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:43.880686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:43.881344 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:43.881348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:43.881495 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:43.882186 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:43.882187 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:43.882228 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:43.882294 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:43.882619 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:43.882631 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:43.882637 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:10:43.882770 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 21:10:43.883061 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260928 21:10:43.885185 9623 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33965
I20260928 21:10:43.885459 9623 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:43.886111 9623 heartbeater.cc:511] Master 127.21.140.126:33965 requested a full tablet report, sending...
I20260928 21:10:43.887593 9435 ts_manager.cc:194] Registered new tserver with Master: 6cea1da0ee884b818fc1fbc6cf6b49ff (127.21.140.65:38173)
I20260928 21:10:43.888828 9435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.65:51677
W20260928 21:10:43.996825 9626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:43.997128 9626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:43.997169 9626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:44.001587 9626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:44.001657 9626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:44.001715 9626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.66
I20260928 21:10:44.006742 9626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.140.66
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.66
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:44.007902 9626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:44.009335 9626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:44.019896 9632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:44.019852 9631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:44.019924 9634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:44.020242 9626 server_base.cc:1034] running on GCE node
I20260928 21:10:44.020740 9626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:44.021561 9626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:44.022789 9626 hybrid_clock.cc:648] HybridClock initialized: now 1790629844022757 us; error 69 us; skew 500 ppm
I20260928 21:10:44.024761 9626 webserver.cc:460] Webserver started at http://127.21.140.66:44835/ using document root <none> and password file <none>
I20260928 21:10:44.025383 9626 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:44.025485 9626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:44.025746 9626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:44.027931 9626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data/instance:
uuid: "a8273affd8e34cc9b8207c1c5b4d5c86"
format_stamp: "Formatted at 2026-09-28 21:10:44 on dist-test-slave-d5wc"
server_key: "a0bd12b2e9bddf7b7771ca6a614fbac4"
server_key_iv: "6a065edaa7ba2828fb23c3572ac0fb36"
server_key_version: "encryptionkey@0"
I20260928 21:10:44.028656 9626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance:
uuid: "a8273affd8e34cc9b8207c1c5b4d5c86"
format_stamp: "Formatted at 2026-09-28 21:10:44 on dist-test-slave-d5wc"
server_key: "a0bd12b2e9bddf7b7771ca6a614fbac4"
server_key_iv: "6a065edaa7ba2828fb23c3572ac0fb36"
server_key_version: "encryptionkey@0"
I20260928 21:10:44.032733 9626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:44.035558 9640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:44.036847 9626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:10:44.037029 9626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal
uuid: "a8273affd8e34cc9b8207c1c5b4d5c86"
format_stamp: "Formatted at 2026-09-28 21:10:44 on dist-test-slave-d5wc"
server_key: "a0bd12b2e9bddf7b7771ca6a614fbac4"
server_key_iv: "6a065edaa7ba2828fb23c3572ac0fb36"
server_key_version: "encryptionkey@0"
I20260928 21:10:44.037166 9626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:44.048552 9626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:44.049291 9626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:44.049511 9626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:44.050169 9626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:44.051151 9626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:44.051229 9626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:44.051302 9626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:44.051347 9626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:44.060388 9626 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.66:43765
I20260928 21:10:44.060439 9753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.66:43765 every 8 connection(s)
I20260928 21:10:44.061368 9626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/data/info.pb
I20260928 21:10:44.071298 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9626
I20260928 21:10:44.071447 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal/instance
I20260928 21:10:44.071455 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.063497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:45453 (local address 127.21.140.126:33965)
0928 21:10:44.063686 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.063692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.064974 (+ 1282us) server_negotiation.cc:407] Connection header received
0928 21:10:44.066470 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.066473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.066527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.066631 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:44.067527 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.068093 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.069245 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.069451 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.069533 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.070874 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:44.070895 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:44.070898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:44.070910 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:44.070931 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:44.070939 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:44.070998 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.071241 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.071295 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 21:10:44.071797 22065 external_mini_cluster.cc:1428] Setting key 8a973898c397f5515d5be0404b6590ee
I20260928 21:10:44.072732 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.064017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33965 (local address 127.21.140.66:45453)
0928 21:10:44.064793 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.064838 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.066220 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.066803 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.066810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.066900 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:44.067398 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.067410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.068216 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.068219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.069131 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.069140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.069662 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.069666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.069833 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.070585 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:44.070587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:44.070632 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:44.070714 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:44.071055 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:44.071067 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:44.071075 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.071251 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.071613 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260928 21:10:44.074105 9754 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33965
I20260928 21:10:44.074193 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.140.67:0
--local_ip_for_outbound_sockets=127.21.140.67
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33965
--builtin_ntp_servers=127.21.140.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:10:44.074429 9754 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:44.075065 9754 heartbeater.cc:511] Master 127.21.140.126:33965 requested a full tablet report, sending...
I20260928 21:10:44.076325 9435 ts_manager.cc:194] Registered new tserver with Master: a8273affd8e34cc9b8207c1c5b4d5c86 (127.21.140.66:43765)
I20260928 21:10:44.077023 9435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.66:45453
W20260928 21:10:44.189126 9757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:44.189426 9757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:44.189466 9757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:44.193725 9757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:44.193792 9757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:44.193852 9757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.140.67
I20260928 21:10:44.198803 9757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.140.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.140.67
--webserver_port=0
--tserver_master_addrs=127.21.140.126:33965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.140.67
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:44.199962 9757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:44.201383 9757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:10:44.210961 9757 server_base.cc:1034] running on GCE node
W20260928 21:10:44.211012 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:44.211128 9762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:44.211021 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:44.211632 9757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:44.212283 9757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:44.213476 9757 hybrid_clock.cc:648] HybridClock initialized: now 1790629844213443 us; error 49 us; skew 500 ppm
I20260928 21:10:44.215449 9757 webserver.cc:460] Webserver started at http://127.21.140.67:35387/ using document root <none> and password file <none>
I20260928 21:10:44.216130 9757 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:44.216199 9757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:44.216364 9757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:44.218426 9757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data/instance:
uuid: "8e7d6da562ba4f44a25994787aa963d3"
format_stamp: "Formatted at 2026-09-28 21:10:44 on dist-test-slave-d5wc"
server_key: "dbcdf9a982b6b72fc39a14876d60072f"
server_key_iv: "8ed87ae7e027a6627473bca4c14c9c50"
server_key_version: "encryptionkey@0"
I20260928 21:10:44.218937 9757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance:
uuid: "8e7d6da562ba4f44a25994787aa963d3"
format_stamp: "Formatted at 2026-09-28 21:10:44 on dist-test-slave-d5wc"
server_key: "dbcdf9a982b6b72fc39a14876d60072f"
server_key_iv: "8ed87ae7e027a6627473bca4c14c9c50"
server_key_version: "encryptionkey@0"
I20260928 21:10:44.222601 9757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:10:44.224946 9771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:44.226224 9757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:44.226343 9757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal
uuid: "8e7d6da562ba4f44a25994787aa963d3"
format_stamp: "Formatted at 2026-09-28 21:10:44 on dist-test-slave-d5wc"
server_key: "dbcdf9a982b6b72fc39a14876d60072f"
server_key_iv: "8ed87ae7e027a6627473bca4c14c9c50"
server_key_version: "encryptionkey@0"
I20260928 21:10:44.226454 9757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:44.241956 9757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:44.242816 9757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:44.243021 9757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:44.243625 9757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:10:44.244570 9757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:10:44.244620 9757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:44.244660 9757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:10:44.244715 9757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:44.254017 9757 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.67:43227
I20260928 21:10:44.254099 9884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.67:43227 every 8 connection(s)
I20260928 21:10:44.254966 9757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/data/info.pb
I20260928 21:10:44.261926 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9757
I20260928 21:10:44.262094 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal/instance
I20260928 21:10:44.262352 22065 external_mini_cluster.cc:1428] Setting key f1e7d383a89c9d05e9b03ead474a2d05
I20260928 21:10:44.264391 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.256811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.67:60869 (local address 127.21.140.126:33965)
0928 21:10:44.256962 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.256967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.258165 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 21:10:44.259514 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.259516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.259564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.259648 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:44.260545 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.261058 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.262319 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.262611 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.262731 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.263886 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:44.263890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:44.263892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:44.263905 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:44.263927 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:44.263943 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:44.264024 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.264194 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.264255 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:10:44.265272 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.257256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:33965 (local address 127.21.140.67:60869)
0928 21:10:44.257954 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.258057 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.259323 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.259813 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.259821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.259904 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:10:44.260395 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.260407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.261172 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.261174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.262194 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.262203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.262777 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.262781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.262950 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.263620 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:10:44.263622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:10:44.263662 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:10:44.263726 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:10:44.264028 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:10:44.264038 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:10:44.264047 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.264190 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.264472 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":106,"threads_started":1}
I20260928 21:10:44.266295 9885 heartbeater.cc:347] Connected to a master server at 127.21.140.126:33965
I20260928 21:10:44.266525 9885 heartbeater.cc:464] Registering TS with master...
I20260928 21:10:44.267023 9885 heartbeater.cc:511] Master 127.21.140.126:33965 requested a full tablet report, sending...
I20260928 21:10:44.267989 9435 ts_manager.cc:194] Registered new tserver with Master: 8e7d6da562ba4f44a25994787aa963d3 (127.21.140.67:43227)
I20260928 21:10:44.268522 9435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.140.67:60869
I20260928 21:10:44.276121 22065 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:10:44.282007 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.278034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.21.140.126:33965)
0928 21:10:44.278196 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.278203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.278376 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:10:44.278511 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.278515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.278566 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.278684 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:44.278975 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.279494 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.280357 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.280602 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.280710 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.281505 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:44.281508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:44.281511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:44.281524 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:44.281551 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:44.281559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:44.281671 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.281769 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.281827 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:10:44.284955 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:10:44.288142 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:10:44.303140 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.296943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:43227 (local address 127.0.0.1:42108)
0928 21:10:44.297946 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.298018 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.298219 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.298879 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.298885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.298905 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:44.299030 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.299035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.300212 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.300216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.301538 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.301548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.302588 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.302592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.302702 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.302754 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.302865 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.302982 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":95,"threads_started":1}
I20260928 21:10:44.303187 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.297091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.21.140.67:43227)
0928 21:10:44.297461 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.297474 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.298106 (+ 632us) server_negotiation.cc:407] Connection header received
0928 21:10:44.298428 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.298434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.298652 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.298812 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:44.299154 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.300087 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.301668 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.302453 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.302565 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.302760 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.302909 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.303010 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":152,"threads_started":1}
I20260928 21:10:44.303454 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.297838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.66:43765 (local address 127.0.0.1:55336)
0928 21:10:44.298652 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.298664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.298767 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.299344 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.299347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.299378 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:44.299541 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.299548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.300725 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.300729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.301803 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.301813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.303042 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.303045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.303138 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.303169 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.303244 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.303314 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":32,"spinlock_wait_cycles":9728,"thread_start_us":89,"threads_started":1}
I20260928 21:10:44.303916 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.298279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.21.140.66:43765)
0928 21:10:44.298593 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.298605 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.298740 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:10:44.298908 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.298912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.299059 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.299202 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:44.299672 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.300606 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.301941 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.302700 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.302800 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.302983 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.303658 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.303762 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260928 21:10:44.303936 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.297148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:38173 (local address 127.0.0.1:51238)
0928 21:10:44.298203 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.298216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.298738 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.299344 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.299347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.299373 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:44.299547 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.299553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.301098 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.301101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.302170 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.302180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.303462 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.303469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.303587 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.303609 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.303725 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.303795 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":36,"spinlock_wait_cycles":2176,"thread_start_us":145,"threads_started":1}
I20260928 21:10:44.304646 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.297640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.21.140.65:38173)
0928 21:10:44.297936 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.297953 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.298680 (+ 727us) server_negotiation.cc:407] Connection header received
0928 21:10:44.298875 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.298881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.299060 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.299202 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:44.299689 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.300971 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.302304 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.303077 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.303195 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.303399 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.303743 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.303832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":122,"threads_started":1}
I20260928 21:10:44.306576 9819 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3637a974e84dbcae9a7edcf67170a0 (DEFAULT_TABLE table=test-table [id=a056c071233b49988be5e964d1cdde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:44.306968 9557 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3637a974e84dbcae9a7edcf67170a0 (DEFAULT_TABLE table=test-table [id=a056c071233b49988be5e964d1cdde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:44.307719 9819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3637a974e84dbcae9a7edcf67170a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:44.307900 9557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3637a974e84dbcae9a7edcf67170a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:44.307885 9688 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3637a974e84dbcae9a7edcf67170a0 (DEFAULT_TABLE table=test-table [id=a056c071233b49988be5e964d1cdde2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:10:44.308779 9688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3637a974e84dbcae9a7edcf67170a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:44.314050 9902 tablet_bootstrap.cc:492] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: Bootstrap starting.
I20260928 21:10:44.315353 9904 tablet_bootstrap.cc:492] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Bootstrap starting.
I20260928 21:10:44.316125 9903 tablet_bootstrap.cc:492] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Bootstrap starting.
I20260928 21:10:44.316419 9902 tablet_bootstrap.cc:654] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:44.317281 9902 log.cc:824] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:44.317499 9904 tablet_bootstrap.cc:654] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:44.318004 9903 tablet_bootstrap.cc:654] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:44.318321 9904 log.cc:824] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:44.318816 9903 log.cc:824] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:44.320212 9902 tablet_bootstrap.cc:492] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: No bootstrap required, opened a new log
I20260928 21:10:44.320420 9902 ts_tablet_manager.cc:1397] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:10:44.320534 9903 tablet_bootstrap.cc:492] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: No bootstrap required, opened a new log
I20260928 21:10:44.320706 9903 ts_tablet_manager.cc:1397] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:10:44.323428 9902 raft_consensus.cc:348] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.323653 9902 raft_consensus.cc:374] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:44.323722 9902 raft_consensus.cc:729] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea1da0ee884b818fc1fbc6cf6b49ff, State: Initialized, Role: FOLLOWER
I20260928 21:10:44.324002 9903 raft_consensus.cc:348] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.324229 9903 raft_consensus.cc:374] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:44.324179 9902 consensus_queue.cc:260] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.324288 9903 raft_consensus.cc:729] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8273affd8e34cc9b8207c1c5b4d5c86, State: Initialized, Role: FOLLOWER
I20260928 21:10:44.324718 9903 consensus_queue.cc:260] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.324795 9623 heartbeater.cc:503] Master 127.21.140.126:33965 was elected leader, sending a full tablet report...
I20260928 21:10:44.325155 9902 ts_tablet_manager.cc:1428] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:10:44.325688 9903 ts_tablet_manager.cc:1428] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:10:44.326272 9904 tablet_bootstrap.cc:492] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: No bootstrap required, opened a new log
I20260928 21:10:44.326469 9904 ts_tablet_manager.cc:1397] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260928 21:10:44.327687 9754 heartbeater.cc:503] Master 127.21.140.126:33965 was elected leader, sending a full tablet report...
I20260928 21:10:44.330394 9904 raft_consensus.cc:348] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.330608 9904 raft_consensus.cc:374] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:44.330641 9904 raft_consensus.cc:729] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d6da562ba4f44a25994787aa963d3, State: Initialized, Role: FOLLOWER
I20260928 21:10:44.330991 9904 consensus_queue.cc:260] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.331655 9885 heartbeater.cc:503] Master 127.21.140.126:33965 was elected leader, sending a full tablet report...
I20260928 21:10:44.331943 9904 ts_tablet_manager.cc:1428] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 21:10:44.374213 9624 tablet.cc:2369] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:10:44.506044 9886 tablet.cc:2369] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:10:44.533649 9909 raft_consensus.cc:482] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:10:44.533876 9909 raft_consensus.cc:504] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.535199 9909 leader_election.cc:288] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e7d6da562ba4f44a25994787aa963d3 (127.21.140.67:43227), 6cea1da0ee884b818fc1fbc6cf6b49ff (127.21.140.65:38173)
I20260928 21:10:44.540073 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.535785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:35261 (local address 127.21.140.65:38173)
0928 21:10:44.535930 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.535936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.535954 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:10:44.536001 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.536004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.536041 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.536144 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:44.536411 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.537141 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.538697 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.539313 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.539441 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.539491 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.539859 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.539938 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:10:44.540073 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.535460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.140.66:34131 (local address 127.21.140.67:43227)
0928 21:10:44.535595 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.535600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.535618 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:10:44.535678 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.535684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.535737 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.535813 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:10:44.536142 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.536979 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.538697 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.539313 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.539441 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.539491 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.539858 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.539945 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:10:44.540117 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.535346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.67:43227 (local address 127.21.140.66:34131)
0928 21:10:44.535473 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.535493 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.535592 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.535838 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.535844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.535891 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:44.536024 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.536030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.537104 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.537107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.538560 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.538573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.539440 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.539444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.539708 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.539744 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.539824 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.539931 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":190,"spinlock_wait_cycles":31872}
I20260928 21:10:44.540125 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.535467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:38173 (local address 127.21.140.66:35261)
0928 21:10:44.535781 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.535797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:10:44.535893 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:10:44.536167 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:10:44.536169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:10:44.536185 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:10:44.536303 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.536307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.537236 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.537239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.538560 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:10:44.538573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.539440 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.539444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:10:44.539682 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.539743 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 21:10:44.539824 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:10:44.539931 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":156,"spinlock_wait_cycles":1792,"thread_start_us":87,"threads_started":1}
I20260928 21:10:44.540841 9577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3637a974e84dbcae9a7edcf67170a0" candidate_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" is_pre_election: true
I20260928 21:10:44.540845 9839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3637a974e84dbcae9a7edcf67170a0" candidate_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d6da562ba4f44a25994787aa963d3" is_pre_election: true
I20260928 21:10:44.541174 9577 raft_consensus.cc:2454] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8273affd8e34cc9b8207c1c5b4d5c86 in term 0.
I20260928 21:10:44.541174 9839 raft_consensus.cc:2454] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8273affd8e34cc9b8207c1c5b4d5c86 in term 0.
I20260928 21:10:44.541712 9642 leader_election.cc:302] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea1da0ee884b818fc1fbc6cf6b49ff, a8273affd8e34cc9b8207c1c5b4d5c86; no voters:
I20260928 21:10:44.542037 9909 raft_consensus.cc:2781] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:10:44.542117 9909 raft_consensus.cc:482] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:10:44.542151 9909 raft_consensus.cc:3037] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:44.543229 9909 raft_consensus.cc:504] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.543664 9909 leader_election.cc:288] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [CANDIDATE]: Term 1 election: Requested vote from peers 8e7d6da562ba4f44a25994787aa963d3 (127.21.140.67:43227), 6cea1da0ee884b818fc1fbc6cf6b49ff (127.21.140.65:38173)
I20260928 21:10:44.544063 9839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3637a974e84dbcae9a7edcf67170a0" candidate_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d6da562ba4f44a25994787aa963d3"
I20260928 21:10:44.544147 9577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3637a974e84dbcae9a7edcf67170a0" candidate_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff"
I20260928 21:10:44.544221 9839 raft_consensus.cc:3037] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:44.544306 9577 raft_consensus.cc:3037] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:44.545382 9839 raft_consensus.cc:2454] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8273affd8e34cc9b8207c1c5b4d5c86 in term 1.
I20260928 21:10:44.545431 9577 raft_consensus.cc:2454] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8273affd8e34cc9b8207c1c5b4d5c86 in term 1.
I20260928 21:10:44.545761 9643 leader_election.cc:302] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d6da562ba4f44a25994787aa963d3, a8273affd8e34cc9b8207c1c5b4d5c86; no voters:
I20260928 21:10:44.545992 9909 raft_consensus.cc:2781] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:44.546239 9909 raft_consensus.cc:686] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 1 LEADER]: Becoming Leader. State: Replica: a8273affd8e34cc9b8207c1c5b4d5c86, State: Running, Role: LEADER
I20260928 21:10:44.546607 9909 consensus_queue.cc:237] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } }
I20260928 21:10:44.549852 9434 catalog_manager.cc:5495] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8273affd8e34cc9b8207c1c5b4d5c86 (127.21.140.66). New cstate: current_term: 1 leader_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8273affd8e34cc9b8207c1c5b4d5c86" member_type: VOTER last_known_addr { host: "127.21.140.66" port: 43765 } health_report { overall_health: HEALTHY } } }
W20260928 21:10:44.562572 9755 tablet.cc:2369] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:10:44.595367 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.591539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.21.140.66:43765)
0928 21:10:44.591714 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:44.591719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:44.591734 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:10:44.591893 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:44.591896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:44.591954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:44.592075 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:10:44.592330 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.592845 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.593793 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:44.594041 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:44.594161 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:44.594491 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:10:44.594957 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:10:44.595095 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:44.595158 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:10:44.595217 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:10:44.601277 9577 raft_consensus.cc:1261] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 1 FOLLOWER]: Refusing update from remote peer a8273affd8e34cc9b8207c1c5b4d5c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:44.601279 9839 raft_consensus.cc:1261] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 1 FOLLOWER]: Refusing update from remote peer a8273affd8e34cc9b8207c1c5b4d5c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:10:44.602082 9909 consensus_queue.cc:1035] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cea1da0ee884b818fc1fbc6cf6b49ff" member_type: VOTER last_known_addr { host: "127.21.140.65" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:44.602303 9912 consensus_queue.cc:1035] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d6da562ba4f44a25994787aa963d3" member_type: VOTER last_known_addr { host: "127.21.140.67" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:10:44.610953 9916 mvcc.cc:205] Tried to move back new op lower bound from 7334419843479646208 to 7334419843266015232. Current Snapshot: MvccSnapshot[applied={T|T < 7334419843479646208}]
I20260928 21:10:44.611960 9917 mvcc.cc:205] Tried to move back new op lower bound from 7334419843479646208 to 7334419843266015232. Current Snapshot: MvccSnapshot[applied={T|T < 7334419843479646208}]
I20260928 21:10:44.628847 9434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:44.629004 9434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:10:44.631476 9434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cec49aba05e64250a4f2209daf7047fb: Sending DeleteTablet for 3 replicas of tablet 0f3637a974e84dbcae9a7edcf67170a0
I20260928 21:10:44.632341 9557 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3637a974e84dbcae9a7edcf67170a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:44 UTC) from {username='slave'} at 127.0.0.1:51238
I20260928 21:10:44.632339 9688 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3637a974e84dbcae9a7edcf67170a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:44 UTC) from {username='slave'} at 127.0.0.1:55336
I20260928 21:10:44.632339 9819 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3637a974e84dbcae9a7edcf67170a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:10:44 UTC) from {username='slave'} at 127.0.0.1:42108
I20260928 21:10:44.632879 9927 tablet_replica.cc:330] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff: stopping tablet replica
I20260928 21:10:44.633023 9926 tablet_replica.cc:330] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: stopping tablet replica
I20260928 21:10:44.633306 9926 raft_consensus.cc:2229] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:44.633306 9927 raft_consensus.cc:2229] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:10:44.633630 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9495
I20260928 21:10:44.633636 9926 raft_consensus.cc:2258] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:44.633636 9927 raft_consensus.cc:2258] T 0f3637a974e84dbcae9a7edcf67170a0 P 6cea1da0ee884b818fc1fbc6cf6b49ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:44.634135 9925 tablet_replica.cc:330] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: stopping tablet replica
I20260928 21:10:44.634415 9925 raft_consensus.cc:2229] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:10:44.634814 9925 raft_consensus.cc:2258] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:10:44.636197 9925 ts_tablet_manager.cc:1905] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:44.636197 9926 ts_tablet_manager.cc:1905] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:10:44.639060 9925 ts_tablet_manager.cc:1918] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:44.639048 9926 ts_tablet_manager.cc:1918] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:10:44.639164 9926 log.cc:1196] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-2/wal/wals/0f3637a974e84dbcae9a7edcf67170a0
I20260928 21:10:44.639302 9925 log.cc:1196] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Deleting WAL directory at /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629583469051-22065-0/minicluster-data/ts-1/wal/wals/0f3637a974e84dbcae9a7edcf67170a0
I20260928 21:10:44.639503 9926 ts_tablet_manager.cc:1939] T 0f3637a974e84dbcae9a7edcf67170a0 P 8e7d6da562ba4f44a25994787aa963d3: Deleting consensus metadata
I20260928 21:10:44.639750 9925 ts_tablet_manager.cc:1939] T 0f3637a974e84dbcae9a7edcf67170a0 P a8273affd8e34cc9b8207c1c5b4d5c86: Deleting consensus metadata
I20260928 21:10:44.640877 9421 catalog_manager.cc:4841] TS a8273affd8e34cc9b8207c1c5b4d5c86 (127.21.140.66:43765): tablet 0f3637a974e84dbcae9a7edcf67170a0 (table test-table [id=a056c071233b49988be5e964d1cdde2e]) successfully deleted
I20260928 21:10:44.641131 9421 catalog_manager.cc:4841] TS 8e7d6da562ba4f44a25994787aa963d3 (127.21.140.67:43227): tablet 0f3637a974e84dbcae9a7edcf67170a0 (table test-table [id=a056c071233b49988be5e964d1cdde2e]) successfully deleted
W20260928 21:10:44.643623 9420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.140.65:38173 (error 108)
I20260928 21:10:44.644469 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9626
I20260928 21:10:44.644740 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:44.644434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.65:38173 (local address 127.0.0.1:56626)
0928 21:10:44.644585 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:10:44.644660 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.65:38173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 21:10:44.645226 9420 catalog_manager.cc:4584] TS 6cea1da0ee884b818fc1fbc6cf6b49ff (127.21.140.65:38173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f3637a974e84dbcae9a7edcf67170a0: Network error: Client connection negotiation failed: client connection to 127.21.140.65:38173: connect: Connection refused (error 111)
I20260928 21:10:44.650601 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9757
I20260928 21:10:44.656155 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9404
2026-09-28T21:10:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1246 ms)
[----------] 6 tests from EncryptionPolicyTest (10662 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:44 dist-test-slave-d5wc krb5kdc[9931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:44 dist-test-slave-d5wc krb5kdc[9931](info): set up 2 sockets
Sep 28 21:10:44 dist-test-slave-d5wc krb5kdc[9931](info): commencing operation
krb5kdc: starting...
W20260928 21:10:45.703913 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:45 dist-test-slave-d5wc krb5kdc[9931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:45.829280 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:45.941252 9944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:45.941537 9944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:45.941588 9944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:45.945572 9944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:45.945640 9944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:45.945659 9944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:45.945675 9944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:45.945690 9944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:45.951007 9944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:45.952054 9944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:45.953006 9944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:45.961936 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:45.961899 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:45.961983 9944 server_base.cc:1034] running on GCE node
W20260928 21:10:45.961899 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:45.962862 9944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:45.964010 9944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:45.965181 9944 hybrid_clock.cc:648] HybridClock initialized: now 1790629845965148 us; error 51 us; skew 500 ppm
Sep 28 21:10:45 dist-test-slave-d5wc krb5kdc[9931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:45.967999 9944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:45.968978 9944 webserver.cc:460] Webserver started at http://127.0.0.1:39753/ using document root <none> and password file <none>
I20260928 21:10:45.969518 9944 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:45.969583 9944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:45.969753 9944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:45.971859 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "27f390c4e07a45a5b6657ee2cc0c28fa"
format_stamp: "Formatted at 2026-09-28 21:10:45 on dist-test-slave-d5wc"
server_key: "20fbc65f1dd0f56ceb471591b5e0e804"
server_key_iv: "dd6eca63efa8301188323c278265c6ba"
server_key_version: "encryptionkey@0"
I20260928 21:10:45.972385 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "27f390c4e07a45a5b6657ee2cc0c28fa"
format_stamp: "Formatted at 2026-09-28 21:10:45 on dist-test-slave-d5wc"
server_key: "20fbc65f1dd0f56ceb471591b5e0e804"
server_key_iv: "dd6eca63efa8301188323c278265c6ba"
server_key_version: "encryptionkey@0"
I20260928 21:10:45.976351 9944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:45.978849 9959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:45.980217 9944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:45.980340 9944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "27f390c4e07a45a5b6657ee2cc0c28fa"
format_stamp: "Formatted at 2026-09-28 21:10:45 on dist-test-slave-d5wc"
server_key: "20fbc65f1dd0f56ceb471591b5e0e804"
server_key_iv: "dd6eca63efa8301188323c278265c6ba"
server_key_version: "encryptionkey@0"
I20260928 21:10:45.980451 9944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:45.992156 9944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:45.995139 9944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:45.995364 9944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:46.002451 9944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32839
I20260928 21:10:46.002524 10011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32839 every 8 connection(s)
I20260928 21:10:46.003497 9944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:46.005605 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 9944
I20260928 21:10:46.005753 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:46.006105 22065 external_mini_cluster.cc:1428] Setting key 0ad1ec7537fadf46c16d3fbb9fcac22e
I20260928 21:10:46.006783 10012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:46.014274 10012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa: Bootstrap starting.
Sep 28 21:10:46 dist-test-slave-d5wc krb5kdc[9931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:10:46.016918 10012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:46.017927 10012 log.cc:824] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:46.020381 10012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa: No bootstrap required, opened a new log
I20260928 21:10:46.022409 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:46.007496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40479 (local address 127.0.0.1:32839)
0928 21:10:46.007982 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:46.008002 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:46.008075 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:10:46.008968 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:46.008991 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:46.009458 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:46.009856 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:46.010692 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:46.012217 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:46.013121 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:46.013813 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:46.014059 (+ 246us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:46.016741 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:46.016748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:46.016769 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:46.016811 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:46.018959 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:46.019542 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:46.019548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:46.019554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:46.019690 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:46.020057 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:46.020060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:46.020062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:46.020594 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:46.020803 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:46.021174 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 21:10:46.021420 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
I20260928 21:10:46.024760 10012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32839 } }
I20260928 21:10:46.025058 10012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:46.025138 10012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f390c4e07a45a5b6657ee2cc0c28fa, State: Initialized, Role: FOLLOWER
I20260928 21:10:46.025653 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:46.025640 10012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32839 } }
I20260928 21:10:46.025811 10012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:46.025872 10012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:46.025996 10012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:46.027318 10012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32839 } }
I20260928 21:10:46.027711 10012 leader_election.cc:302] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27f390c4e07a45a5b6657ee2cc0c28fa; no voters:
I20260928 21:10:46.028115 10012 leader_election.cc:288] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:46.028220 10021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:46.028525 10021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [term 1 LEADER]: Becoming Leader. State: Replica: 27f390c4e07a45a5b6657ee2cc0c28fa, State: Running, Role: LEADER
I20260928 21:10:46.028874 10021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32839 } }
I20260928 21:10:46.029728 10012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:46.030620 10023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27f390c4e07a45a5b6657ee2cc0c28fa. Latest consensus state: current_term: 1 leader_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32839 } } }
I20260928 21:10:46.030757 10023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:46.031070 10024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f390c4e07a45a5b6657ee2cc0c28fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32839 } } }
I20260928 21:10:46.031219 10024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:46.031522 10031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:46.035203 10031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:46.038362 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:46.028102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36447 (local address 127.0.0.1:32839)
0928 21:10:46.028230 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:46.028237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:46.028604 (+ 367us) server_negotiation.cc:407] Connection header received
0928 21:10:46.028805 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:46.028808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:46.028856 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:46.028906 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:46.029909 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:46.030776 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:46.031969 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:46.032297 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:46.032407 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:46.034379 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:46.034383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:46.034386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:46.034415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:46.036151 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:46.036758 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:46.036762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:46.036764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:46.036859 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:46.037308 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:46.037316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:46.037317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:46.037495 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:46.037664 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:46.037943 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 21:10:46.038074 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:10:46.041743 10031 catalog_manager.cc:1347] Generated new cluster ID: e269b0e2facb4969af779b490403fd43
I20260928 21:10:46.041813 10031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:46.084192 10031 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:46.085529 10031 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:46.093634 10031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27f390c4e07a45a5b6657ee2cc0c28fa: Generated new TSK 0
I20260928 21:10:46.094504 10031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:46.104800 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 9944
W20260928 21:10:46.110244 9929 connection.cc:535] client connection to 127.0.0.1:32839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32839): Connection reset by peer (error 104)
2026-09-28T21:10:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:46 dist-test-slave-d5wc krb5kdc[10043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:46 dist-test-slave-d5wc krb5kdc[10043](info): set up 2 sockets
Sep 28 21:10:46 dist-test-slave-d5wc krb5kdc[10043](info): commencing operation
krb5kdc: starting...
W20260928 21:10:47.139694 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:47 dist-test-slave-d5wc krb5kdc[10043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:47.269277 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:47.384261 10056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:47.384620 10056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:47.384701 10056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:47.389088 10056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:47.389212 10056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:47.389249 10056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:47.389288 10056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:47.389323 10056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:47.395126 10056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:47.396258 10056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:47.397306 10056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:47.406149 10062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:47.406173 10064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:47.406154 10061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:47.406545 10056 server_base.cc:1034] running on GCE node
I20260928 21:10:47.407189 10056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:47.408406 10056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:47.409581 10056 hybrid_clock.cc:648] HybridClock initialized: now 1790629847409553 us; error 48 us; skew 500 ppm
Sep 28 21:10:47 dist-test-slave-d5wc krb5kdc[10043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:47.418622 10056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:47.419669 10056 webserver.cc:460] Webserver started at http://127.0.0.1:34903/ using document root <none> and password file <none>
I20260928 21:10:47.420230 10056 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:47.420295 10056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:47.420485 10056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:47.422648 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "c92537936dbf4887abcb56e3a8736e5d"
format_stamp: "Formatted at 2026-09-28 21:10:47 on dist-test-slave-d5wc"
server_key: "c8628f3f7e14a08701670c1c5a7b9eb8"
server_key_iv: "52cbd478d746f9b6e22f86e4fe9e8475"
server_key_version: "encryptionkey@0"
I20260928 21:10:47.423197 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "c92537936dbf4887abcb56e3a8736e5d"
format_stamp: "Formatted at 2026-09-28 21:10:47 on dist-test-slave-d5wc"
server_key: "c8628f3f7e14a08701670c1c5a7b9eb8"
server_key_iv: "52cbd478d746f9b6e22f86e4fe9e8475"
server_key_version: "encryptionkey@0"
I20260928 21:10:47.427000 10056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:10:47.429394 10071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:47.430755 10056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:47.430855 10056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "c92537936dbf4887abcb56e3a8736e5d"
format_stamp: "Formatted at 2026-09-28 21:10:47 on dist-test-slave-d5wc"
server_key: "c8628f3f7e14a08701670c1c5a7b9eb8"
server_key_iv: "52cbd478d746f9b6e22f86e4fe9e8475"
server_key_version: "encryptionkey@0"
I20260928 21:10:47.430940 10056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:47.445886 10056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:47.449175 10056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:47.449334 10056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:47.456414 10056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33517
I20260928 21:10:47.456430 10123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33517 every 8 connection(s)
I20260928 21:10:47.457494 10056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:47.460309 10124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:47.465770 10124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d: Bootstrap starting.
I20260928 21:10:47.466130 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10056
I20260928 21:10:47.466262 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:47.466547 22065 external_mini_cluster.cc:1428] Setting key e248a515543e8aad2b4d26367051b492
I20260928 21:10:47.468568 10124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:47.469326 10124 log.cc:824] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:47.471359 10124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d: No bootstrap required, opened a new log
I20260928 21:10:47.475672 10124 raft_consensus.cc:348] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92537936dbf4887abcb56e3a8736e5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33517 } }
I20260928 21:10:47.475847 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:47.467901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42519 (local address 127.0.0.1:33517)
0928 21:10:47.468333 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:47.468348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:47.468381 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:10:47.469082 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:47.469128 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:47.469515 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:47.469863 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:47.470295 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:47.471245 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:47.472465 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:47.472954 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:47.473078 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:47.474073 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:47.474078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:47.474117 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:47.474501 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:47.474554 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:47.474578 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:47.474695 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:47.474886 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 21:10:47.475049 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260928 21:10:47.475900 10124 raft_consensus.cc:374] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:47.475935 10124 raft_consensus.cc:729] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92537936dbf4887abcb56e3a8736e5d, State: Initialized, Role: FOLLOWER
I20260928 21:10:47.476421 10124 consensus_queue.cc:260] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92537936dbf4887abcb56e3a8736e5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33517 } }
I20260928 21:10:47.476550 10124 raft_consensus.cc:388] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:47.476637 10124 raft_consensus.cc:482] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:47.476763 10124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:47.477949 10124 raft_consensus.cc:504] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92537936dbf4887abcb56e3a8736e5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33517 } }
I20260928 21:10:47.478327 10124 leader_election.cc:302] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c92537936dbf4887abcb56e3a8736e5d; no voters:
I20260928 21:10:47.478672 10124 leader_election.cc:288] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:47.478740 10129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:47.478957 10129 raft_consensus.cc:686] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [term 1 LEADER]: Becoming Leader. State: Replica: c92537936dbf4887abcb56e3a8736e5d, State: Running, Role: LEADER
I20260928 21:10:47.478988 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:47.479326 10129 consensus_queue.cc:237] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92537936dbf4887abcb56e3a8736e5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33517 } }
I20260928 21:10:47.479952 10124 sys_catalog.cc:564] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:47.481009 10131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c92537936dbf4887abcb56e3a8736e5d. Latest consensus state: current_term: 1 leader_uuid: "c92537936dbf4887abcb56e3a8736e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92537936dbf4887abcb56e3a8736e5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33517 } } }
I20260928 21:10:47.481145 10131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:47.481458 10130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c92537936dbf4887abcb56e3a8736e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92537936dbf4887abcb56e3a8736e5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33517 } } }
I20260928 21:10:47.481559 10130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:47.481667 10143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:47.484969 10143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:47.486941 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:47.481347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56307 (local address 127.0.0.1:33517)
0928 21:10:47.481449 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:47.481455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:47.482419 (+ 964us) server_negotiation.cc:407] Connection header received
0928 21:10:47.482459 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:47.482462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:47.482506 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:47.482583 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:47.482977 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:47.483723 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:47.484844 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:47.485217 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:47.485322 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:47.486385 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:47.486389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:47.486391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:47.486405 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:47.486434 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:47.486448 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:47.486535 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:47.486769 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 21:10:47.486809 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:10:47.491374 10143 catalog_manager.cc:1347] Generated new cluster ID: 38674def2dd2451082a15659efd3fe9b
I20260928 21:10:47.491448 10143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:47.496465 10143 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:47.497349 10143 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:47.507828 10143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c92537936dbf4887abcb56e3a8736e5d: Generated new TSK 0
I20260928 21:10:47.508580 10143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:47.512063 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10056
W20260928 21:10:47.517633 10041 connection.cc:535] client connection to 127.0.0.1:33517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33517): Connection reset by peer (error 104)
2026-09-28T21:10:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:47 dist-test-slave-d5wc krb5kdc[10155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:47 dist-test-slave-d5wc krb5kdc[10155](info): set up 2 sockets
Sep 28 21:10:47 dist-test-slave-d5wc krb5kdc[10155](info): commencing operation
krb5kdc: starting...
W20260928 21:10:48.567719 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:48 dist-test-slave-d5wc krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:48.690718 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:48.800858 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:48.801183 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:48.801265 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:48.805395 10168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:48.805482 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:48.805516 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:48.805553 10168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:48.805590 10168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:48.811038 10168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:48.812187 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:48.813198 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:10:48.821883 10168 server_base.cc:1034] running on GCE node
W20260928 21:10:48.821870 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:48.821874 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:48.822113 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:48.822693 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:48.823843 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:48.825026 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790629848825014 us; error 27 us; skew 500 ppm
Sep 28 21:10:48 dist-test-slave-d5wc krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:48.827726 10168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:48.828684 10168 webserver.cc:460] Webserver started at http://127.0.0.1:42863/ using document root <none> and password file <none>
I20260928 21:10:48.829286 10168 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:48.829377 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:48.829612 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:48.831729 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "2638343896f84fe49ea4a3e315503150"
format_stamp: "Formatted at 2026-09-28 21:10:48 on dist-test-slave-d5wc"
server_key: "8efb51891a2b61278952caf124394f3b"
server_key_iv: "5e88538ae7351daa6cff8d87c8016e8e"
server_key_version: "encryptionkey@0"
I20260928 21:10:48.832335 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "2638343896f84fe49ea4a3e315503150"
format_stamp: "Formatted at 2026-09-28 21:10:48 on dist-test-slave-d5wc"
server_key: "8efb51891a2b61278952caf124394f3b"
server_key_iv: "5e88538ae7351daa6cff8d87c8016e8e"
server_key_version: "encryptionkey@0"
I20260928 21:10:48.836146 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:10:48.838681 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:48.839982 10168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:48.840139 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "2638343896f84fe49ea4a3e315503150"
format_stamp: "Formatted at 2026-09-28 21:10:48 on dist-test-slave-d5wc"
server_key: "8efb51891a2b61278952caf124394f3b"
server_key_iv: "5e88538ae7351daa6cff8d87c8016e8e"
server_key_version: "encryptionkey@0"
I20260928 21:10:48.840256 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:48.844913 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:48.847908 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:48.848106 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:48.855067 10168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39061
I20260928 21:10:48.855089 10235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39061 every 8 connection(s)
I20260928 21:10:48.855999 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:48.856564 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10168
I20260928 21:10:48.856760 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:48.857044 22065 external_mini_cluster.cc:1428] Setting key a4d17ba330014b0da378e0db0e136511
I20260928 21:10:48.859174 10236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:48.862234 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:48.858443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53291 (local address 127.0.0.1:39061)
0928 21:10:48.858889 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:48.858905 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:48.858939 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:10:48.859653 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:48.859671 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:48.859948 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:48.860291 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:48.861023 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:48.861029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:48.861043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:48.861059 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:48.861108 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:48.861127 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:48.861275 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:48.861345 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:10:48.861491 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":151,"threads_started":1}
I20260928 21:10:48.864948 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:48.865455 10236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150: Bootstrap starting.
I20260928 21:10:48.868003 10236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:48.868801 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:48.867093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51481 (local address 127.0.0.1:39061)
0928 21:10:48.867226 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:48.867230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:48.867404 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:10:48.867521 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:48.867523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:48.867554 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:48.867636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:48.868409 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:48.868413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:48.868415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:48.868427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:48.868454 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:48.868471 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:48.868554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:48.868645 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:10:48.868685 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:10:48.868772 10236 log.cc:824] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:48.870859 10236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150: No bootstrap required, opened a new log
I20260928 21:10:48.874485 10236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638343896f84fe49ea4a3e315503150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260928 21:10:48.874761 10236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:48.874832 10236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2638343896f84fe49ea4a3e315503150, State: Initialized, Role: FOLLOWER
I20260928 21:10:48.875355 10236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638343896f84fe49ea4a3e315503150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260928 21:10:48.875536 10236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:48.875605 10236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:48.875717 10236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:48.876816 10236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638343896f84fe49ea4a3e315503150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260928 21:10:48.877195 10236 leader_election.cc:302] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2638343896f84fe49ea4a3e315503150; no voters:
I20260928 21:10:48.877545 10236 leader_election.cc:288] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:48.877676 10246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:48.877905 10246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [term 1 LEADER]: Becoming Leader. State: Replica: 2638343896f84fe49ea4a3e315503150, State: Running, Role: LEADER
I20260928 21:10:48.878881 10236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:48.878944 10246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638343896f84fe49ea4a3e315503150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } }
I20260928 21:10:48.880597 10248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2638343896f84fe49ea4a3e315503150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638343896f84fe49ea4a3e315503150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } } }
I20260928 21:10:48.880748 10248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:48.881064 10247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2638343896f84fe49ea4a3e315503150. Latest consensus state: current_term: 1 leader_uuid: "2638343896f84fe49ea4a3e315503150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2638343896f84fe49ea4a3e315503150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39061 } } }
I20260928 21:10:48.881484 10247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150 [sys.catalog]: This master's current role is: LEADER
W20260928 21:10:48.883606 10261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:10:48.883692 10261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:10:48.883773 10260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:48.884443 10260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:48.889707 10260 catalog_manager.cc:1347] Generated new cluster ID: e1a70d8612f64093ab1785124e1e57f5
I20260928 21:10:48.889772 10260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:48.906365 10260 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:48.906720 10260 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:48.916278 10260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2638343896f84fe49ea4a3e315503150: Generated new TSK 0
I20260928 21:10:48.917114 10260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:48.930428 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10168
2026-09-28T21:10:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:48 dist-test-slave-d5wc krb5kdc[10267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:48 dist-test-slave-d5wc krb5kdc[10267](info): set up 2 sockets
Sep 28 21:10:48 dist-test-slave-d5wc krb5kdc[10267](info): commencing operation
krb5kdc: starting...
W20260928 21:10:49.995760 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:50 dist-test-slave-d5wc krb5kdc[10267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:50.125665 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:50.236605 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:50.236877 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:50.236958 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:50.241181 10280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:50.241243 10280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:50.241263 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:50.241281 10280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:50.241343 10280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:50.246757 10280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:50.247828 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:50.248859 10280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:50.257513 10286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:50.257504 10285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:50.257553 10280 server_base.cc:1034] running on GCE node
W20260928 21:10:50.257504 10288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:50.258373 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:50.259457 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:50.260627 10280 hybrid_clock.cc:648] HybridClock initialized: now 1790629850260599 us; error 41 us; skew 500 ppm
Sep 28 21:10:50 dist-test-slave-d5wc krb5kdc[10267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:50.263554 10280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:50.264508 10280 webserver.cc:460] Webserver started at http://127.0.0.1:36941/ using document root <none> and password file <none>
I20260928 21:10:50.265089 10280 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:50.265152 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:50.265399 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:50.267505 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "b414a9dc30a04e2bbdb1bf64aba55043"
format_stamp: "Formatted at 2026-09-28 21:10:50 on dist-test-slave-d5wc"
server_key: "408037b83499384c5602644bcf3f138c"
server_key_iv: "22c6dfd7c907c5a902f16656287ad14a"
server_key_version: "encryptionkey@0"
I20260928 21:10:50.268096 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "b414a9dc30a04e2bbdb1bf64aba55043"
format_stamp: "Formatted at 2026-09-28 21:10:50 on dist-test-slave-d5wc"
server_key: "408037b83499384c5602644bcf3f138c"
server_key_iv: "22c6dfd7c907c5a902f16656287ad14a"
server_key_version: "encryptionkey@0"
I20260928 21:10:50.271957 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:50.274482 10295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:50.275758 10280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:50.275888 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "b414a9dc30a04e2bbdb1bf64aba55043"
format_stamp: "Formatted at 2026-09-28 21:10:50 on dist-test-slave-d5wc"
server_key: "408037b83499384c5602644bcf3f138c"
server_key_iv: "22c6dfd7c907c5a902f16656287ad14a"
server_key_version: "encryptionkey@0"
I20260928 21:10:50.275993 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:50.296693 10280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:50.299793 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:50.299996 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:50.307337 10280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33651
I20260928 21:10:50.307361 10347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33651 every 8 connection(s)
I20260928 21:10:50.308676 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:50.311491 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:50.312014 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10280
I20260928 21:10:50.312178 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:50.312469 22065 external_mini_cluster.cc:1428] Setting key 6aaa1d921eb312667c284e61e51539a6
I20260928 21:10:50.318064 10348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043: Bootstrap starting.
I20260928 21:10:50.320374 10348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:50.321271 10348 log.cc:824] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:10:50 dist-test-slave-d5wc krb5kdc[10267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:10:50.323747 10348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043: No bootstrap required, opened a new log
I20260928 21:10:50.326629 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:50.313857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52797 (local address 127.0.0.1:33651)
0928 21:10:50.314331 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:50.314353 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:50.314388 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:10:50.315093 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:50.315114 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:50.315449 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:50.315829 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:50.316640 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:50.317600 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:50.318577 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:50.319049 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:50.319222 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:50.322109 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:50.322115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:50.322129 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:50.322154 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:50.323842 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:50.324287 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:50.324294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:50.324304 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:50.324413 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:50.324690 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:50.324695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:50.324697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:50.325074 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:50.325290 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:50.325562 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 21:10:50.325852 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":132,"threads_started":1}
I20260928 21:10:50.327386 10348 raft_consensus.cc:348] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260928 21:10:50.327643 10348 raft_consensus.cc:374] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:50.327700 10348 raft_consensus.cc:729] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b414a9dc30a04e2bbdb1bf64aba55043, State: Initialized, Role: FOLLOWER
I20260928 21:10:50.328157 10348 consensus_queue.cc:260] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260928 21:10:50.328308 10348 raft_consensus.cc:388] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:50.328377 10348 raft_consensus.cc:482] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:50.328490 10348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:50.329465 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:50.329591 10348 raft_consensus.cc:504] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260928 21:10:50.329926 10348 leader_election.cc:302] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b414a9dc30a04e2bbdb1bf64aba55043; no voters:
I20260928 21:10:50.330292 10348 leader_election.cc:288] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:50.330374 10353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:50.330544 10353 raft_consensus.cc:686] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [term 1 LEADER]: Becoming Leader. State: Replica: b414a9dc30a04e2bbdb1bf64aba55043, State: Running, Role: LEADER
I20260928 21:10:50.330838 10353 consensus_queue.cc:237] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260928 21:10:50.331558 10348 sys_catalog.cc:564] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:50.332449 10354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } } }
I20260928 21:10:50.332578 10354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:50.332928 10355 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b414a9dc30a04e2bbdb1bf64aba55043. Latest consensus state: current_term: 1 leader_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414a9dc30a04e2bbdb1bf64aba55043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } } }
I20260928 21:10:50.333020 10355 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:50.333916 10367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:50.336925 10367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:50.341017 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:50.331675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37319 (local address 127.0.0.1:33651)
0928 21:10:50.331791 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:50.331795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:50.332145 (+ 350us) server_negotiation.cc:407] Connection header received
0928 21:10:50.332360 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:50.332364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:50.332422 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:50.332477 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:50.333474 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:50.334336 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:50.335466 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:50.335795 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:50.335907 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:50.337550 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:50.337553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:50.337556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:50.337583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:50.339328 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:50.339779 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:50.339784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:50.339786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:50.339865 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:50.340159 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:50.340162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:50.340164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:50.340344 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:50.340488 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:50.340730 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 21:10:50.340844 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:10:50.346853 10367 catalog_manager.cc:1347] Generated new cluster ID: c60daaec947b4bbcbd7715be709b522d
I20260928 21:10:50.346943 10367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:50.366499 10367 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:50.367847 10367 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:50.379355 10367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b414a9dc30a04e2bbdb1bf64aba55043: Generated new TSK 0
I20260928 21:10:50.380265 10367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:50.385282 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10280
2026-09-28T21:10:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:50 dist-test-slave-d5wc krb5kdc[10379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:50 dist-test-slave-d5wc krb5kdc[10379](info): set up 2 sockets
Sep 28 21:10:50 dist-test-slave-d5wc krb5kdc[10379](info): commencing operation
krb5kdc: starting...
W20260928 21:10:51.447710 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:51 dist-test-slave-d5wc krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:51.571182 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:51.683414 10392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:51.683691 10392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:51.683734 10392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:51.687909 10392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:51.687975 10392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:51.687995 10392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:51.688012 10392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:51.688027 10392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:51.693317 10392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:51.694427 10392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:51.695453 10392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:51.704043 10400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:51.704046 10398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:51.704046 10397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:51.704286 10392 server_base.cc:1034] running on GCE node
I20260928 21:10:51.705051 10392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:51.706188 10392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:51.707368 10392 hybrid_clock.cc:648] HybridClock initialized: now 1790629851707338 us; error 40 us; skew 500 ppm
Sep 28 21:10:51 dist-test-slave-d5wc krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:51.710152 10392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:51.710978 10392 webserver.cc:460] Webserver started at http://127.0.0.1:33337/ using document root <none> and password file <none>
I20260928 21:10:51.711499 10392 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:51.711557 10392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:51.711723 10392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:51.713716 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "57b4fa81a13a4a7cafe267d9a272be17"
format_stamp: "Formatted at 2026-09-28 21:10:51 on dist-test-slave-d5wc"
server_key: "f27ff781cbbda9aa5892591cfe0472e8"
server_key_iv: "f2ef871200ca9c6ed738f001d54f7685"
server_key_version: "encryptionkey@0"
I20260928 21:10:51.714278 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "57b4fa81a13a4a7cafe267d9a272be17"
format_stamp: "Formatted at 2026-09-28 21:10:51 on dist-test-slave-d5wc"
server_key: "f27ff781cbbda9aa5892591cfe0472e8"
server_key_iv: "f2ef871200ca9c6ed738f001d54f7685"
server_key_version: "encryptionkey@0"
I20260928 21:10:51.718151 10392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 21:10:51.720554 10407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:51.721835 10392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:10:51.722041 10392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "57b4fa81a13a4a7cafe267d9a272be17"
format_stamp: "Formatted at 2026-09-28 21:10:51 on dist-test-slave-d5wc"
server_key: "f27ff781cbbda9aa5892591cfe0472e8"
server_key_iv: "f2ef871200ca9c6ed738f001d54f7685"
server_key_version: "encryptionkey@0"
I20260928 21:10:51.722148 10392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:51.732560 10392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:51.735374 10392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:51.735534 10392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:51.742326 10392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38325
I20260928 21:10:51.742336 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38325 every 8 connection(s)
I20260928 21:10:51.743294 10392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:51.746040 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:51.747664 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10392
I20260928 21:10:51.747826 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:51.748081 22065 external_mini_cluster.cc:1428] Setting key d855ddabe197838072b87336d42e58c2
I20260928 21:10:51.752795 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17: Bootstrap starting.
I20260928 21:10:51.755417 10460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:51.756155 10460 log.cc:824] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:51.758096 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:51.749472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39255 (local address 127.0.0.1:38325)
0928 21:10:51.749907 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:51.749926 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:51.749989 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:10:51.750865 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:51.750890 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:51.751321 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:51.751706 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:51.752113 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:51.753545 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:51.754607 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:51.755279 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:51.755448 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:51.756396 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:51.756401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:51.756414 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:51.756809 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:51.756863 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:51.756886 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:51.757009 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:51.757128 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:10:51.757333 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
I20260928 21:10:51.758198 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17: No bootstrap required, opened a new log
I20260928 21:10:51.761003 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:51.762362 10460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b4fa81a13a4a7cafe267d9a272be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260928 21:10:51.762584 10460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:51.762655 10460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b4fa81a13a4a7cafe267d9a272be17, State: Initialized, Role: FOLLOWER
I20260928 21:10:51.763136 10460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b4fa81a13a4a7cafe267d9a272be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260928 21:10:51.763309 10460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:51.763383 10460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:51.763484 10460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:51.764398 10460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b4fa81a13a4a7cafe267d9a272be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260928 21:10:51.764727 10460 leader_election.cc:302] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b4fa81a13a4a7cafe267d9a272be17; no voters:
I20260928 21:10:51.765043 10460 leader_election.cc:288] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:51.765112 10470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:51.765326 10470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [term 1 LEADER]: Becoming Leader. State: Replica: 57b4fa81a13a4a7cafe267d9a272be17, State: Running, Role: LEADER
I20260928 21:10:51.765730 10470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b4fa81a13a4a7cafe267d9a272be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260928 21:10:51.766398 10460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:51.767452 10472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57b4fa81a13a4a7cafe267d9a272be17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b4fa81a13a4a7cafe267d9a272be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } } }
I20260928 21:10:51.767596 10472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:51.767802 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:51.763073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54655 (local address 127.0.0.1:38325)
0928 21:10:51.763194 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:51.763198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:51.763349 (+ 151us) server_negotiation.cc:407] Connection header received
0928 21:10:51.763526 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:51.763529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:51.763629 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:51.763717 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:51.763964 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:51.764504 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:51.765597 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:51.765887 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:51.766013 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:51.766952 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:51.766956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:51.766959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:51.766971 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:51.766998 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:51.767010 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:51.767074 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:51.767502 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 21:10:51.767627 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:10:51.767922 10471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b4fa81a13a4a7cafe267d9a272be17. Latest consensus state: current_term: 1 leader_uuid: "57b4fa81a13a4a7cafe267d9a272be17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b4fa81a13a4a7cafe267d9a272be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } } }
I20260928 21:10:51.768100 10471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17 [sys.catalog]: This master's current role is: LEADER
W20260928 21:10:51.771134 10485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:10:51.771206 10485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:10:51.771278 10479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:51.772142 10479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:51.777576 10479 catalog_manager.cc:1347] Generated new cluster ID: 50f5fdee705b44f396756dd6afdf187f
I20260928 21:10:51.777642 10479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:51.806365 10479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:51.807663 10479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:51.813530 10479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57b4fa81a13a4a7cafe267d9a272be17: Generated new TSK 0
I20260928 21:10:51.814385 10479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:51.826742 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10392
2026-09-28T21:10:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:51 dist-test-slave-d5wc krb5kdc[10491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:51 dist-test-slave-d5wc krb5kdc[10491](info): set up 2 sockets
Sep 28 21:10:51 dist-test-slave-d5wc krb5kdc[10491](info): commencing operation
krb5kdc: starting...
W20260928 21:10:52.875782 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:52 dist-test-slave-d5wc krb5kdc[10491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:53.001355 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:53.112187 10504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:53.112491 10504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:53.112582 10504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:53.116667 10504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:53.116771 10504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:53.116811 10504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:53.116845 10504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:53.116879 10504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:53.122447 10504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:53.123514 10504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:53.124585 10504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:53.133431 10512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:53.133375 10510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:53.133466 10504 server_base.cc:1034] running on GCE node
W20260928 21:10:53.133417 10509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:53.134227 10504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:53.135344 10504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:53.136554 10504 hybrid_clock.cc:648] HybridClock initialized: now 1790629853136536 us; error 31 us; skew 500 ppm
Sep 28 21:10:53 dist-test-slave-d5wc krb5kdc[10491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:53.143625 10504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:53.144608 10504 webserver.cc:460] Webserver started at http://127.0.0.1:42425/ using document root <none> and password file <none>
I20260928 21:10:53.145185 10504 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:53.145277 10504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:53.145503 10504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:53.147643 10504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "f52cadf4327b460f90f84a91a5725a2d"
format_stamp: "Formatted at 2026-09-28 21:10:53 on dist-test-slave-d5wc"
server_key: "b2b7464d20dc9f9484be69d545fb1d0f"
server_key_iv: "9cc3fec6f90e8ed883f4688bcce7090e"
server_key_version: "encryptionkey@0"
I20260928 21:10:53.148209 10504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "f52cadf4327b460f90f84a91a5725a2d"
format_stamp: "Formatted at 2026-09-28 21:10:53 on dist-test-slave-d5wc"
server_key: "b2b7464d20dc9f9484be69d545fb1d0f"
server_key_iv: "9cc3fec6f90e8ed883f4688bcce7090e"
server_key_version: "encryptionkey@0"
I20260928 21:10:53.152133 10504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:10:53.154692 10519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:53.156034 10504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:10:53.156184 10504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "f52cadf4327b460f90f84a91a5725a2d"
format_stamp: "Formatted at 2026-09-28 21:10:53 on dist-test-slave-d5wc"
server_key: "b2b7464d20dc9f9484be69d545fb1d0f"
server_key_iv: "9cc3fec6f90e8ed883f4688bcce7090e"
server_key_version: "encryptionkey@0"
I20260928 21:10:53.156301 10504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:53.160972 10504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:53.163833 10504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:53.164024 10504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:53.171124 10504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38923
I20260928 21:10:53.171154 10571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38923 every 8 connection(s)
I20260928 21:10:53.172077 10504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:53.174854 10572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:53.177758 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10504
I20260928 21:10:53.177938 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:53.178257 22065 external_mini_cluster.cc:1428] Setting key 989d6c670af6b5beae9443ff6fd13725
I20260928 21:10:53.181149 10572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d: Bootstrap starting.
I20260928 21:10:53.183610 10572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:53.183733 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:53.179737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42147 (local address 127.0.0.1:38923)
0928 21:10:53.180229 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:53.180245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:53.180282 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:10:53.180970 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:53.180987 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:53.181336 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:53.181686 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:53.182495 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:53.182501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:53.182518 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:53.182529 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:53.182586 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:53.182606 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:53.182753 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:53.182828 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:10:53.182993 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
I20260928 21:10:53.184384 10572 log.cc:824] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:53.186568 10572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d: No bootstrap required, opened a new log
I20260928 21:10:53.186870 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:53.190742 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:53.189049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57611 (local address 127.0.0.1:38923)
0928 21:10:53.189192 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:53.189198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:53.189436 (+ 238us) server_negotiation.cc:407] Connection header received
0928 21:10:53.189533 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:53.189536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:53.189573 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:53.189663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:53.190406 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:53.190409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:53.190412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:53.190423 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:53.190451 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:53.190463 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:53.190557 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:53.190594 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 21:10:53.190632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:10:53.190593 10572 raft_consensus.cc:348] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f52cadf4327b460f90f84a91a5725a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260928 21:10:53.190872 10572 raft_consensus.cc:374] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:53.190939 10572 raft_consensus.cc:729] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52cadf4327b460f90f84a91a5725a2d, State: Initialized, Role: FOLLOWER
I20260928 21:10:53.191413 10572 consensus_queue.cc:260] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f52cadf4327b460f90f84a91a5725a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260928 21:10:53.191586 10572 raft_consensus.cc:388] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:53.191653 10572 raft_consensus.cc:482] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:53.191756 10572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:53.192723 10572 raft_consensus.cc:504] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f52cadf4327b460f90f84a91a5725a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260928 21:10:53.193014 10572 leader_election.cc:302] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f52cadf4327b460f90f84a91a5725a2d; no voters:
I20260928 21:10:53.193326 10572 leader_election.cc:288] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:53.193436 10582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:53.193660 10582 raft_consensus.cc:686] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [term 1 LEADER]: Becoming Leader. State: Replica: f52cadf4327b460f90f84a91a5725a2d, State: Running, Role: LEADER
I20260928 21:10:53.194032 10582 consensus_queue.cc:237] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f52cadf4327b460f90f84a91a5725a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } }
I20260928 21:10:53.194700 10572 sys_catalog.cc:564] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:53.195833 10583 sys_catalog.cc:455] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f52cadf4327b460f90f84a91a5725a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f52cadf4327b460f90f84a91a5725a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } } }
I20260928 21:10:53.195957 10583 sys_catalog.cc:458] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:53.196242 10584 sys_catalog.cc:455] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f52cadf4327b460f90f84a91a5725a2d. Latest consensus state: current_term: 1 leader_uuid: "f52cadf4327b460f90f84a91a5725a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f52cadf4327b460f90f84a91a5725a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38923 } } }
I20260928 21:10:53.196319 10584 sys_catalog.cc:458] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:53.196380 10591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:53.199646 10591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:53.205509 10591 catalog_manager.cc:1347] Generated new cluster ID: b992e1a4109545e8ab162d15c871b660
I20260928 21:10:53.205597 10591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:53.222667 10591 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:53.223047 10591 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:53.235394 10591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f52cadf4327b460f90f84a91a5725a2d: Generated new TSK 0
I20260928 21:10:53.236270 10591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:53.248678 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10504
2026-09-28T21:10:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:53 dist-test-slave-d5wc krb5kdc[10603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:53 dist-test-slave-d5wc krb5kdc[10603](info): set up 2 sockets
Sep 28 21:10:53 dist-test-slave-d5wc krb5kdc[10603](info): commencing operation
krb5kdc: starting...
W20260928 21:10:54.303905 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:54 dist-test-slave-d5wc krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:54.434324 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:54.545722 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:54.546025 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:54.546068 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:54.550275 10616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:54.550340 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:54.550361 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:54.550379 10616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:54.550395 10616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:54.555763 10616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:54.556849 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:54.557876 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:54.566936 10622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:54.566936 10621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:54.566988 10616 server_base.cc:1034] running on GCE node
W20260928 21:10:54.567021 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:54.567850 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:54.568985 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:54.570152 10616 hybrid_clock.cc:648] HybridClock initialized: now 1790629854570134 us; error 40 us; skew 500 ppm
Sep 28 21:10:54 dist-test-slave-d5wc krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:54.572926 10616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:54.573864 10616 webserver.cc:460] Webserver started at http://127.0.0.1:44127/ using document root <none> and password file <none>
I20260928 21:10:54.574456 10616 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:54.574524 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:54.574738 10616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:54.576844 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "e14196c909134ee282307af9ab9e7803"
format_stamp: "Formatted at 2026-09-28 21:10:54 on dist-test-slave-d5wc"
server_key: "d5c1bf99bef972c72292a872ed9de6a7"
server_key_iv: "101efb4755b40365e024aa0a2c4080fa"
server_key_version: "encryptionkey@0"
I20260928 21:10:54.577425 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "e14196c909134ee282307af9ab9e7803"
format_stamp: "Formatted at 2026-09-28 21:10:54 on dist-test-slave-d5wc"
server_key: "d5c1bf99bef972c72292a872ed9de6a7"
server_key_iv: "101efb4755b40365e024aa0a2c4080fa"
server_key_version: "encryptionkey@0"
I20260928 21:10:54.581274 10616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:54.583930 10631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:54.585256 10616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:10:54.585404 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "e14196c909134ee282307af9ab9e7803"
format_stamp: "Formatted at 2026-09-28 21:10:54 on dist-test-slave-d5wc"
server_key: "d5c1bf99bef972c72292a872ed9de6a7"
server_key_iv: "101efb4755b40365e024aa0a2c4080fa"
server_key_version: "encryptionkey@0"
I20260928 21:10:54.585528 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:54.597500 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:54.600934 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:54.601176 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:54.608386 10616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36043
I20260928 21:10:54.608451 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36043 every 8 connection(s)
I20260928 21:10:54.609396 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:54.610957 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10616
I20260928 21:10:54.611143 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:54.611457 22065 external_mini_cluster.cc:1428] Setting key ffeb95b394d358ed08b88258c7b7cc8d
I20260928 21:10:54.612949 10684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:54.620108 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803: Bootstrap starting.
Sep 28 21:10:54 dist-test-slave-d5wc krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:10:54.622725 10684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:54.623483 10684 log.cc:824] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:54.625677 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803: No bootstrap required, opened a new log
I20260928 21:10:54.627310 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:54.612882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:36125 (local address 127.0.0.1:36043)
0928 21:10:54.613359 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:54.613375 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:54.613412 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:10:54.614134 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:54.614154 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:54.614506 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:54.614805 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:54.615797 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:54.616906 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:54.618171 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:54.618666 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:54.618793 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:54.621795 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:54.621801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:54.621814 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:54.621847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:54.623882 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:54.624562 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:54.624569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:54.624573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:54.624697 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:54.625037 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:54.625040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:54.625042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:54.625458 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:54.625717 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:54.626036 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 21:10:54.626304 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":199,"threads_started":1}
I20260928 21:10:54.629824 10684 raft_consensus.cc:348] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14196c909134ee282307af9ab9e7803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 21:10:54.630127 10684 raft_consensus.cc:374] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:54.630227 10684 raft_consensus.cc:729] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14196c909134ee282307af9ab9e7803, State: Initialized, Role: FOLLOWER
I20260928 21:10:54.630674 10684 consensus_queue.cc:260] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14196c909134ee282307af9ab9e7803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 21:10:54.630805 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:54.630838 10684 raft_consensus.cc:388] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:54.630903 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:54.630995 10684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:54.632330 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14196c909134ee282307af9ab9e7803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 21:10:54.632705 10684 leader_election.cc:302] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e14196c909134ee282307af9ab9e7803; no voters:
I20260928 21:10:54.633100 10684 leader_election.cc:288] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:54.633168 10693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:54.633425 10693 raft_consensus.cc:686] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [term 1 LEADER]: Becoming Leader. State: Replica: e14196c909134ee282307af9ab9e7803, State: Running, Role: LEADER
I20260928 21:10:54.633790 10693 consensus_queue.cc:237] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14196c909134ee282307af9ab9e7803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 21:10:54.634519 10684 sys_catalog.cc:564] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:54.635142 10693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e14196c909134ee282307af9ab9e7803. Latest consensus state: current_term: 1 leader_uuid: "e14196c909134ee282307af9ab9e7803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14196c909134ee282307af9ab9e7803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } } }
I20260928 21:10:54.635928 10693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:54.635818 10695 sys_catalog.cc:455] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e14196c909134ee282307af9ab9e7803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14196c909134ee282307af9ab9e7803" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } } }
I20260928 21:10:54.636266 10695 sys_catalog.cc:458] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803 [sys.catalog]: This master's current role is: LEADER
W20260928 21:10:54.639991 10709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:10:54.640090 10709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:10:54.640167 10703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:54.640910 10703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:54.643143 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:54.633211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:38029 (local address 127.0.0.1:36043)
0928 21:10:54.633331 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:54.633336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:54.633575 (+ 239us) server_negotiation.cc:407] Connection header received
0928 21:10:54.633754 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:54.633758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:54.633811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:54.633893 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:54.634628 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:54.635289 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:54.636442 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:54.636776 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:54.636873 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:54.639020 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:54.639025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:54.639028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:54.639059 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:54.640906 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:54.641592 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:54.641595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:54.641597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:54.641689 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:54.642136 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:54.642140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:54.642141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:54.642364 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:54.642533 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:54.642861 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 21:10:54.642973 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:10:54.646929 10703 catalog_manager.cc:1347] Generated new cluster ID: 165cbab0c8304d2c963a5ea586ff3998
I20260928 21:10:54.647017 10703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:54.657791 10703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:54.659188 10703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:54.669524 10703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e14196c909134ee282307af9ab9e7803: Generated new TSK 0
I20260928 21:10:54.670496 10703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:54.680387 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10616
W20260928 21:10:54.685380 10601 connection.cc:535] client connection to 127.0.0.1:36043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36043): Connection reset by peer (error 104)
2026-09-28T21:10:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:54 dist-test-slave-d5wc krb5kdc[10715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:54 dist-test-slave-d5wc krb5kdc[10715](info): set up 2 sockets
Sep 28 21:10:54 dist-test-slave-d5wc krb5kdc[10715](info): commencing operation
krb5kdc: starting...
W20260928 21:10:55.727742 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:55 dist-test-slave-d5wc krb5kdc[10715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:55.853425 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:55.964387 10728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:55.964704 10728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:55.964775 10728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:55.968815 10728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:55.968916 10728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:55.968954 10728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:55.969003 10728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:55.969043 10728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:55.974315 10728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:55.975394 10728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:55.976418 10728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:55.984920 10734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:55.984933 10733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:55.984979 10728 server_base.cc:1034] running on GCE node
W20260928 21:10:55.985072 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:55.985737 10728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:55.986917 10728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:55.988083 10728 hybrid_clock.cc:648] HybridClock initialized: now 1790629855988070 us; error 26 us; skew 500 ppm
Sep 28 21:10:55 dist-test-slave-d5wc krb5kdc[10715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:55.991110 10728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:55.992022 10728 webserver.cc:460] Webserver started at http://127.0.0.1:45745/ using document root <none> and password file <none>
I20260928 21:10:55.992565 10728 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:55.992651 10728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:55.992879 10728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:55.995172 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "fba6574db4e540e3933a68bdfa4c8d45"
format_stamp: "Formatted at 2026-09-28 21:10:55 on dist-test-slave-d5wc"
server_key: "36ea0034a2bb7f8fa223a701441b3266"
server_key_iv: "f69fc3712474ea3f802342ff637004b4"
server_key_version: "encryptionkey@0"
I20260928 21:10:55.995769 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "fba6574db4e540e3933a68bdfa4c8d45"
format_stamp: "Formatted at 2026-09-28 21:10:55 on dist-test-slave-d5wc"
server_key: "36ea0034a2bb7f8fa223a701441b3266"
server_key_iv: "f69fc3712474ea3f802342ff637004b4"
server_key_version: "encryptionkey@0"
I20260928 21:10:55.999639 10728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:10:56.002077 10743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:56.003333 10728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:10:56.003510 10728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "fba6574db4e540e3933a68bdfa4c8d45"
format_stamp: "Formatted at 2026-09-28 21:10:55 on dist-test-slave-d5wc"
server_key: "36ea0034a2bb7f8fa223a701441b3266"
server_key_iv: "f69fc3712474ea3f802342ff637004b4"
server_key_version: "encryptionkey@0"
I20260928 21:10:56.003619 10728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:56.023664 10728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:56.027271 10728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:56.027554 10728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:56.035286 10728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43673
I20260928 21:10:56.035310 10795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43673 every 8 connection(s)
I20260928 21:10:56.036327 10728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:56.039266 10796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:56.039703 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10728
I20260928 21:10:56.039849 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:56.040148 22065 external_mini_cluster.cc:1428] Setting key 1cc02a1e889155a588098d2b6e31184c
I20260928 21:10:56.046249 10796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45: Bootstrap starting.
I20260928 21:10:56.048529 10796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:56.049377 10796 log.cc:824] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:56.049419 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:56.041600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:55263 (local address 127.0.0.1:43673)
0928 21:10:56.042114 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:56.042136 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:56.042173 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:10:56.042850 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:56.042869 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:56.043239 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:56.043602 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:56.044015 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:56.045006 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:56.045932 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:56.046553 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:56.046736 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:56.047731 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:56.047736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:56.047749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:56.048156 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:56.048212 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:56.048236 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:56.048378 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:56.048530 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 21:10:56.048669 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":146,"threads_started":1}
I20260928 21:10:56.051456 10796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45: No bootstrap required, opened a new log
I20260928 21:10:56.052394 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:56.055632 10796 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba6574db4e540e3933a68bdfa4c8d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 21:10:56.055905 10796 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:56.055966 10796 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba6574db4e540e3933a68bdfa4c8d45, State: Initialized, Role: FOLLOWER
I20260928 21:10:56.056432 10796 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba6574db4e540e3933a68bdfa4c8d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 21:10:56.056595 10796 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:56.056679 10796 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:56.056782 10796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:56.057924 10796 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba6574db4e540e3933a68bdfa4c8d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 21:10:56.058334 10796 leader_election.cc:302] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba6574db4e540e3933a68bdfa4c8d45; no voters:
I20260928 21:10:56.058702 10796 leader_election.cc:288] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:56.058797 10806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:56.058969 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:56.054517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:50777 (local address 127.0.0.1:43673)
0928 21:10:56.054659 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:56.054665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:56.054830 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:10:56.054981 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:56.054985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:56.055033 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:56.055115 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:56.055397 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:56.055953 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:56.056798 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:56.057190 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:56.057302 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:56.058233 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:56.058237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:56.058239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:56.058252 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:56.058286 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:56.058297 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:56.058396 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:56.058705 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 21:10:56.058744 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:10:56.059044 10806 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [term 1 LEADER]: Becoming Leader. State: Replica: fba6574db4e540e3933a68bdfa4c8d45, State: Running, Role: LEADER
I20260928 21:10:56.059402 10806 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba6574db4e540e3933a68bdfa4c8d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 21:10:56.060460 10796 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:56.061120 10807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba6574db4e540e3933a68bdfa4c8d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba6574db4e540e3933a68bdfa4c8d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } } }
I20260928 21:10:56.061249 10807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:56.061677 10812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:56.061662 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba6574db4e540e3933a68bdfa4c8d45. Latest consensus state: current_term: 1 leader_uuid: "fba6574db4e540e3933a68bdfa4c8d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba6574db4e540e3933a68bdfa4c8d45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } } }
I20260928 21:10:56.062541 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:56.064960 10812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:56.071007 10812 catalog_manager.cc:1347] Generated new cluster ID: 12640bf664fb432ba309469ed156de21
I20260928 21:10:56.071143 10812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:56.083317 10812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:56.084602 10812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:56.095793 10812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fba6574db4e540e3933a68bdfa4c8d45: Generated new TSK 0
I20260928 21:10:56.096727 10812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:56.109557 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10728
W20260928 21:10:56.114852 10713 connection.cc:535] client connection to 127.0.0.1:43673 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43673): Connection reset by peer (error 104)
2026-09-28T21:10:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:56 dist-test-slave-d5wc krb5kdc[10827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:56 dist-test-slave-d5wc krb5kdc[10827](info): set up 2 sockets
Sep 28 21:10:56 dist-test-slave-d5wc krb5kdc[10827](info): commencing operation
krb5kdc: starting...
W20260928 21:10:57.151678 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:57 dist-test-slave-d5wc krb5kdc[10827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:57.277556 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:57.399122 10840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:57.399461 10840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:57.399562 10840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:57.403847 10840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:57.403956 10840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:57.403995 10840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:57.404035 10840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:57.404073 10840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:57.410264 10840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:57.411489 10840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:57.412583 10840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:57.421763 10846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:57.421767 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:57.421763 10845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:57.422119 10840 server_base.cc:1034] running on GCE node
I20260928 21:10:57.422828 10840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:57.424041 10840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:57.425244 10840 hybrid_clock.cc:648] HybridClock initialized: now 1790629857425222 us; error 43 us; skew 500 ppm
Sep 28 21:10:57 dist-test-slave-d5wc krb5kdc[10827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:57.428200 10840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:57.429229 10840 webserver.cc:460] Webserver started at http://127.0.0.1:35597/ using document root <none> and password file <none>
I20260928 21:10:57.429793 10840 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:57.429857 10840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:57.430100 10840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:57.432411 10840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4"
format_stamp: "Formatted at 2026-09-28 21:10:57 on dist-test-slave-d5wc"
server_key: "34c1ff5fb6b5041da2a9d6dd12901799"
server_key_iv: "a56659a7d5bcb923347337d0ec64b1ac"
server_key_version: "encryptionkey@0"
I20260928 21:10:57.432953 10840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4"
format_stamp: "Formatted at 2026-09-28 21:10:57 on dist-test-slave-d5wc"
server_key: "34c1ff5fb6b5041da2a9d6dd12901799"
server_key_iv: "a56659a7d5bcb923347337d0ec64b1ac"
server_key_version: "encryptionkey@0"
I20260928 21:10:57.436865 10840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:10:57.439548 10855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:57.440905 10840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:10:57.441011 10840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4"
format_stamp: "Formatted at 2026-09-28 21:10:57 on dist-test-slave-d5wc"
server_key: "34c1ff5fb6b5041da2a9d6dd12901799"
server_key_iv: "a56659a7d5bcb923347337d0ec64b1ac"
server_key_version: "encryptionkey@0"
I20260928 21:10:57.441107 10840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:57.445761 10840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:57.448849 10840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:57.449010 10840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:57.456390 10840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34083
I20260928 21:10:57.456413 10907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34083 every 8 connection(s)
I20260928 21:10:57.457372 10840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:57.460222 10908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:57.464401 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10840
I20260928 21:10:57.464561 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:57.464859 22065 external_mini_cluster.cc:1428] Setting key 1eebd5759c9f2e378883fcf738ba3db3
I20260928 21:10:57.466560 10908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4: Bootstrap starting.
I20260928 21:10:57.468905 10908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:57.469702 10908 log.cc:824] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:57.470598 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:57.466397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:38873 (local address 127.0.0.1:34083)
0928 21:10:57.466889 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:57.466904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:57.466939 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:10:57.467575 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:57.467591 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:57.467877 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:57.468256 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:57.469049 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:57.469055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:57.469072 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:57.469084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:57.469143 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:57.469170 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:57.469288 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:57.469393 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:10:57.469628 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":144,"threads_started":1}
I20260928 21:10:57.472020 10908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4: No bootstrap required, opened a new log
I20260928 21:10:57.473459 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:57.476910 10908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260928 21:10:57.477231 10908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:57.477330 10908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bdfb2eaf1a4a49b1093346b7b8e3f4, State: Initialized, Role: FOLLOWER
I20260928 21:10:57.477501 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:57.475769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:45007 (local address 127.0.0.1:34083)
0928 21:10:57.475902 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:57.475907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:57.476097 (+ 190us) server_negotiation.cc:407] Connection header received
0928 21:10:57.476227 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:57.476230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:57.476263 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:57.476329 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:57.477051 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:57.477054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:57.477056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:10:57.477066 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:57.477089 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:10:57.477100 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:57.477174 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:57.477302 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 21:10:57.477351 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:10:57.477867 10908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260928 21:10:57.478076 10908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:57.478147 10908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:57.478248 10908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:57.479244 10908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260928 21:10:57.479599 10908 leader_election.cc:302] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83bdfb2eaf1a4a49b1093346b7b8e3f4; no voters:
I20260928 21:10:57.479915 10908 leader_election.cc:288] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:57.480058 10918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:57.480296 10918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 83bdfb2eaf1a4a49b1093346b7b8e3f4, State: Running, Role: LEADER
I20260928 21:10:57.480683 10918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260928 21:10:57.481199 10908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:57.482530 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260928 21:10:57.482544 10920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83bdfb2eaf1a4a49b1093346b7b8e3f4. Latest consensus state: current_term: 1 leader_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bdfb2eaf1a4a49b1093346b7b8e3f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260928 21:10:57.482671 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:57.482685 10920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:57.483072 10927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:57.486014 10927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:57.491839 10927 catalog_manager.cc:1347] Generated new cluster ID: eb874a06a2d14e4eae3ec04d0c895630
I20260928 21:10:57.491918 10927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:57.506986 10927 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:57.507401 10927 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:57.517068 10927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83bdfb2eaf1a4a49b1093346b7b8e3f4: Generated new TSK 0
I20260928 21:10:57.517943 10927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:57.521315 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10840
2026-09-28T21:10:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:57 dist-test-slave-d5wc krb5kdc[10939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:57 dist-test-slave-d5wc krb5kdc[10939](info): set up 2 sockets
Sep 28 21:10:57 dist-test-slave-d5wc krb5kdc[10939](info): commencing operation
krb5kdc: starting...
W20260928 21:10:58.551712 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:10:58 dist-test-slave-d5wc krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:10:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:10:58.677893 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:10:58.787284 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:10:58.787623 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:10:58.787714 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:10:58.791785 10952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:10:58.791888 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:10:58.791927 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:10:58.791965 10952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:10:58.791996 10952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:10:58.797406 10952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:10:58.798556 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:10:58.799605 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:10:58.808343 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:58.808425 10952 server_base.cc:1034] running on GCE node
W20260928 21:10:58.808308 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:10:58.808301 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:10:58.809144 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:10:58.810348 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:10:58.811548 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790629858811519 us; error 42 us; skew 500 ppm
Sep 28 21:10:58 dist-test-slave-d5wc krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:10:58.818292 10952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:10:58.819257 10952 webserver.cc:460] Webserver started at http://127.0.0.1:46003/ using document root <none> and password file <none>
I20260928 21:10:58.819864 10952 fs_manager.cc:359] Metadata directory not provided
I20260928 21:10:58.819962 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:10:58.820204 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:10:58.822383 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c"
format_stamp: "Formatted at 2026-09-28 21:10:58 on dist-test-slave-d5wc"
server_key: "ac5628b5503982d11381edca466b7d75"
server_key_iv: "511955a30c37d4a38d4795513b21ec06"
server_key_version: "encryptionkey@0"
I20260928 21:10:58.822966 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c"
format_stamp: "Formatted at 2026-09-28 21:10:58 on dist-test-slave-d5wc"
server_key: "ac5628b5503982d11381edca466b7d75"
server_key_iv: "511955a30c37d4a38d4795513b21ec06"
server_key_version: "encryptionkey@0"
I20260928 21:10:58.826786 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:10:58.829211 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:10:58.830623 10952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:10:58.830798 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c"
format_stamp: "Formatted at 2026-09-28 21:10:58 on dist-test-slave-d5wc"
server_key: "ac5628b5503982d11381edca466b7d75"
server_key_iv: "511955a30c37d4a38d4795513b21ec06"
server_key_version: "encryptionkey@0"
I20260928 21:10:58.830921 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:10:58.842410 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:10:58.845284 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:10:58.845502 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:10:58.852515 10952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34403
I20260928 21:10:58.852543 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34403 every 8 connection(s)
I20260928 21:10:58.853485 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:10:58.854176 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 10952
I20260928 21:10:58.854347 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:10:58.854647 22065 external_mini_cluster.cc:1428] Setting key 867c029f7a13a8fb39abc7e06c41575f
I20260928 21:10:58.856559 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:10:58.863229 11020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c: Bootstrap starting.
Sep 28 21:10:58 dist-test-slave-d5wc krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:10:58.865875 11020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c: Neither blocks nor log segments found. Creating new log.
I20260928 21:10:58.866849 11020 log.cc:824] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:10:58.869235 11020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c: No bootstrap required, opened a new log
I20260928 21:10:58.870440 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:58.856079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:44485 (local address 127.0.0.1:34403)
0928 21:10:58.856487 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:58.856505 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:58.856548 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:10:58.857371 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:58.857393 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:58.857820 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:58.858224 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:58.858899 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:58.860407 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:58.861281 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:58.862010 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:58.862195 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:58.865050 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:58.865056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:58.865079 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:58.865112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:58.867188 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:58.867848 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:58.867854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:58.867859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:58.867987 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:58.868298 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:58.868301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:58.868303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:58.868796 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:58.868996 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:58.869247 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 21:10:58.869478 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260928 21:10:58.873044 11020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260928 21:10:58.873299 11020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:10:58.873363 11020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee7d77956ab4e06a9ec41bf55e3e04c, State: Initialized, Role: FOLLOWER
I20260928 21:10:58.873809 11020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260928 21:10:58.874009 11020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:10:58.874102 11020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:10:58.874130 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:10:58.874207 11020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:10:58.875417 11020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260928 21:10:58.875809 11020 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee7d77956ab4e06a9ec41bf55e3e04c; no voters:
I20260928 21:10:58.876142 11020 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:10:58.876211 11029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:10:58.876492 11029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [term 1 LEADER]: Becoming Leader. State: Replica: 1ee7d77956ab4e06a9ec41bf55e3e04c, State: Running, Role: LEADER
I20260928 21:10:58.876801 11029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260928 21:10:58.877611 11020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:10:58.878015 11029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee7d77956ab4e06a9ec41bf55e3e04c. Latest consensus state: current_term: 1 leader_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260928 21:10:58.878134 11029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [sys.catalog]: This master's current role is: LEADER
I20260928 21:10:58.878321 11029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d77956ab4e06a9ec41bf55e3e04c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260928 21:10:58.878404 11029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c [sys.catalog]: This master's current role is: LEADER
W20260928 21:10:58.882570 11045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:10:58.882731 11045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:10:58.882887 11034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:10:58.884049 11034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:10:58.885512 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:10:58.876439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:56513 (local address 127.0.0.1:34403)
0928 21:10:58.876586 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:10:58.876591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:10:58.876847 (+ 256us) server_negotiation.cc:407] Connection header received
0928 21:10:58.877015 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:10:58.877018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:10:58.877073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:10:58.877163 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:10:58.877871 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:58.878678 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:58.879864 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:10:58.880231 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:10:58.880338 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:10:58.882133 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:10:58.882136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:10:58.882139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:10:58.882175 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:10:58.883678 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:58.884198 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:58.884204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:58.884206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:58.884292 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:10:58.884639 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:10:58.884642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:10:58.884644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:10:58.884824 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:10:58.884976 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:10:58.885257 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 21:10:58.885351 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:10:58.889635 11034 catalog_manager.cc:1347] Generated new cluster ID: 104970287eec4de7994e98db601e3b3c
I20260928 21:10:58.889705 11034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:10:58.915627 11034 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:10:58.916899 11034 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:10:58.926587 11034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee7d77956ab4e06a9ec41bf55e3e04c: Generated new TSK 0
I20260928 21:10:58.927582 11034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:10:58.939103 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 10952
W20260928 21:10:58.944078 10937 connection.cc:535] client connection to 127.0.0.1:34403 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34403): Connection reset by peer (error 104)
2026-09-28T21:10:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:10:58 dist-test-slave-d5wc krb5kdc[11051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:10:58 dist-test-slave-d5wc krb5kdc[11051](info): set up 2 sockets
Sep 28 21:10:58 dist-test-slave-d5wc krb5kdc[11051](info): commencing operation
krb5kdc: starting...
W20260928 21:11:00.007807 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:00 dist-test-slave-d5wc krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:11:00.136922 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:00.246022 11064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:00.246356 11064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:00.246433 11064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:00.250494 11064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:00.250597 11064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:00.250638 11064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:00.250697 11064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:00.250725 11064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:00.256414 11064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:00.257582 11064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:00.258697 11064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:00.268077 11072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:00.268064 11069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:00.268141 11064 server_base.cc:1034] running on GCE node
W20260928 21:11:00.268064 11070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:00.268880 11064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:00.270080 11064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:00.271293 11064 hybrid_clock.cc:648] HybridClock initialized: now 1790629860271277 us; error 39 us; skew 500 ppm
Sep 28 21:11:00 dist-test-slave-d5wc krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:00.274211 11064 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:11:00.275098 11064 webserver.cc:460] Webserver started at http://127.0.0.1:36029/ using document root <none> and password file <none>
I20260928 21:11:00.275640 11064 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:00.275697 11064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:00.275862 11064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:00.277835 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "e12606aacaf647809bbf700cc438aef4"
format_stamp: "Formatted at 2026-09-28 21:11:00 on dist-test-slave-d5wc"
server_key: "53c4525dde69ab25feb6862527ac2035"
server_key_iv: "29d7f162b5cb1f5d8d8d78f4abe037e2"
server_key_version: "encryptionkey@0"
I20260928 21:11:00.278425 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "e12606aacaf647809bbf700cc438aef4"
format_stamp: "Formatted at 2026-09-28 21:11:00 on dist-test-slave-d5wc"
server_key: "53c4525dde69ab25feb6862527ac2035"
server_key_iv: "29d7f162b5cb1f5d8d8d78f4abe037e2"
server_key_version: "encryptionkey@0"
I20260928 21:11:00.282194 11064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:11:00.284651 11079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:00.285908 11064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:00.286101 11064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "e12606aacaf647809bbf700cc438aef4"
format_stamp: "Formatted at 2026-09-28 21:11:00 on dist-test-slave-d5wc"
server_key: "53c4525dde69ab25feb6862527ac2035"
server_key_iv: "29d7f162b5cb1f5d8d8d78f4abe037e2"
server_key_version: "encryptionkey@0"
I20260928 21:11:00.286212 11064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:00.305040 11064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:00.308152 11064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:00.308369 11064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:00.315634 11064 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34207
I20260928 21:11:00.315653 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34207 every 8 connection(s)
I20260928 21:11:00.316720 11064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:00.319552 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:00.323478 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11064
I20260928 21:11:00.323634 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:00.323932 22065 external_mini_cluster.cc:1428] Setting key 79ee7877f443810fd49cac0f0d860a1f
I20260928 21:11:00.325996 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4: Bootstrap starting.
I20260928 21:11:00.328876 11132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:00.329775 11132 log.cc:824] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:00.332197 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4: No bootstrap required, opened a new log
I20260928 21:11:00.335065 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:00.325452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:37763 (local address 127.0.0.1:34207)
0928 21:11:00.325912 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:00.325931 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:00.326007 (+ 76us) server_negotiation.cc:407] Connection header received
0928 21:11:00.326865 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:00.326889 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:00.327378 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:00.327759 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:00.328221 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:00.329749 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:00.331015 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:00.331694 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:00.331856 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:00.332723 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:00.332730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:00.332747 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:00.333266 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:00.333338 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:00.333369 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:00.333502 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:00.333668 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:11:00.333806 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260928 21:11:00.336361 11132 raft_consensus.cc:348] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12606aacaf647809bbf700cc438aef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34207 } }
I20260928 21:11:00.336591 11132 raft_consensus.cc:374] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:00.336660 11132 raft_consensus.cc:729] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12606aacaf647809bbf700cc438aef4, State: Initialized, Role: FOLLOWER
I20260928 21:11:00.337175 11132 consensus_queue.cc:260] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12606aacaf647809bbf700cc438aef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34207 } }
I20260928 21:11:00.337340 11132 raft_consensus.cc:388] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:00.337433 11132 raft_consensus.cc:482] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:00.337555 11132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:00.338192 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:00.338822 11132 raft_consensus.cc:504] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12606aacaf647809bbf700cc438aef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34207 } }
I20260928 21:11:00.339190 11132 leader_election.cc:302] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12606aacaf647809bbf700cc438aef4; no voters:
I20260928 21:11:00.339591 11132 leader_election.cc:288] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:00.339630 11139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:00.339923 11139 raft_consensus.cc:686] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [term 1 LEADER]: Becoming Leader. State: Replica: e12606aacaf647809bbf700cc438aef4, State: Running, Role: LEADER
I20260928 21:11:00.340329 11139 consensus_queue.cc:237] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12606aacaf647809bbf700cc438aef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34207 } }
I20260928 21:11:00.341041 11132 sys_catalog.cc:564] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:00.341696 11143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12606aacaf647809bbf700cc438aef4. Latest consensus state: current_term: 1 leader_uuid: "e12606aacaf647809bbf700cc438aef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12606aacaf647809bbf700cc438aef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34207 } } }
I20260928 21:11:00.341912 11143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:00.342156 11142 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e12606aacaf647809bbf700cc438aef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12606aacaf647809bbf700cc438aef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34207 } } }
I20260928 21:11:00.342258 11142 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:00.342644 11151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:00.345678 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:00.340464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:52989 (local address 127.0.0.1:34207)
0928 21:11:00.340574 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:00.340579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:00.340817 (+ 238us) server_negotiation.cc:407] Connection header received
0928 21:11:00.340962 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:00.340965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:00.341017 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:00.341081 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:00.341495 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:00.342328 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:00.343679 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:00.344031 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:00.344138 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:00.345190 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:00.345193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:00.345196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:00.345217 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:00.345243 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:00.345254 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:00.345362 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:00.345496 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 21:11:00.345532 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:11:00.346007 11151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:00.352094 11151 catalog_manager.cc:1347] Generated new cluster ID: 2b06b621da6f46ebb28701483f338198
I20260928 21:11:00.352191 11151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:00.368328 11151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:00.369661 11151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:00.374804 11151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e12606aacaf647809bbf700cc438aef4: Generated new TSK 0
I20260928 21:11:00.375528 11151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:00.385931 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11064
W20260928 21:11:00.391700 11049 connection.cc:535] client connection to 127.0.0.1:34207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34207): Connection reset by peer (error 104)
2026-09-28T21:11:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:00 dist-test-slave-d5wc krb5kdc[11163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:00 dist-test-slave-d5wc krb5kdc[11163](info): set up 2 sockets
Sep 28 21:11:00 dist-test-slave-d5wc krb5kdc[11163](info): commencing operation
krb5kdc: starting...
W20260928 21:11:01.431742 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:01 dist-test-slave-d5wc krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:11:01.559118 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:01.669948 11176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:01.670320 11176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:01.670405 11176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:01.674494 11176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:01.674594 11176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:01.674633 11176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:01.674685 11176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:01.674719 11176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:01.680119 11176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:01.681232 11176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:01.682297 11176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:01.690762 11181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:01.690752 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:01.690768 11182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:01.690901 11176 server_base.cc:1034] running on GCE node
I20260928 21:11:01.691576 11176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:01.692646 11176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:01.693804 11176 hybrid_clock.cc:648] HybridClock initialized: now 1790629861693778 us; error 40 us; skew 500 ppm
Sep 28 21:11:01 dist-test-slave-d5wc krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:01.696610 11176 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:11:01.697443 11176 webserver.cc:460] Webserver started at http://127.0.0.1:35313/ using document root <none> and password file <none>
I20260928 21:11:01.697956 11176 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:01.698094 11176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:01.698266 11176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:01.700225 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "c1c7b769b2ef46cca82906894d964652"
format_stamp: "Formatted at 2026-09-28 21:11:01 on dist-test-slave-d5wc"
server_key: "d06356182bdc17e76ed8f8407edc0837"
server_key_iv: "57ee663713fe2a3c4dc2d87994eb1cc5"
server_key_version: "encryptionkey@0"
I20260928 21:11:01.700768 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "c1c7b769b2ef46cca82906894d964652"
format_stamp: "Formatted at 2026-09-28 21:11:01 on dist-test-slave-d5wc"
server_key: "d06356182bdc17e76ed8f8407edc0837"
server_key_iv: "57ee663713fe2a3c4dc2d87994eb1cc5"
server_key_version: "encryptionkey@0"
I20260928 21:11:01.704804 11176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:11:01.707290 11191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:01.708531 11176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:11:01.708662 11176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "c1c7b769b2ef46cca82906894d964652"
format_stamp: "Formatted at 2026-09-28 21:11:01 on dist-test-slave-d5wc"
server_key: "d06356182bdc17e76ed8f8407edc0837"
server_key_iv: "57ee663713fe2a3c4dc2d87994eb1cc5"
server_key_version: "encryptionkey@0"
I20260928 21:11:01.708771 11176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:01.713307 11176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:01.716151 11176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:01.716351 11176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:01.723078 11176 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33641
I20260928 21:11:01.723097 11243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33641 every 8 connection(s)
I20260928 21:11:01.723986 11176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:01.725060 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11176
I20260928 21:11:01.725236 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:01.725524 22065 external_mini_cluster.cc:1428] Setting key fa497c3201f63dcd44f2d26a54f6221d
I20260928 21:11:01.727049 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:01.732899 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652: Bootstrap starting.
I20260928 21:11:01.735193 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:01.726970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:49727 (local address 127.0.0.1:33641)
0928 21:11:01.727391 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:01.727407 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:01.727441 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:11:01.728056 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:01.728073 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:01.728338 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:01.728683 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:01.733899 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:01.733904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:01.733916 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:01.733928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:01.734042 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:01.734068 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:01.734224 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:01.734311 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:11:01.734482 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":129,"threads_started":1}
I20260928 21:11:01.735433 11244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:01.736258 11244 log.cc:824] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:01.738312 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:01.738691 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652: No bootstrap required, opened a new log
I20260928 21:11:01.742432 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:01.740477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:57621 (local address 127.0.0.1:33641)
0928 21:11:01.740651 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:01.740656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:01.740755 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:11:01.740912 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:01.740915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:01.740956 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:01.741044 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:01.741906 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:01.741909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:01.741912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:01.741926 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:01.741959 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:01.742003 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:01.742126 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:01.742198 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:11:01.742254 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 21:11:01.743571 11244 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7b769b2ef46cca82906894d964652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260928 21:11:01.743872 11244 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:01.743917 11244 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c7b769b2ef46cca82906894d964652, State: Initialized, Role: FOLLOWER
I20260928 21:11:01.744313 11244 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7b769b2ef46cca82906894d964652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260928 21:11:01.744431 11244 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:01.744477 11244 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:01.744551 11244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:01.745522 11244 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7b769b2ef46cca82906894d964652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260928 21:11:01.745886 11244 leader_election.cc:302] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c7b769b2ef46cca82906894d964652; no voters:
I20260928 21:11:01.746270 11244 leader_election.cc:288] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:01.746399 11254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:01.746639 11254 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [term 1 LEADER]: Becoming Leader. State: Replica: c1c7b769b2ef46cca82906894d964652, State: Running, Role: LEADER
I20260928 21:11:01.747014 11254 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7b769b2ef46cca82906894d964652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260928 21:11:01.747716 11244 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:01.748857 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c7b769b2ef46cca82906894d964652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7b769b2ef46cca82906894d964652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } } }
I20260928 21:11:01.748982 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:01.749382 11263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:01.749346 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c7b769b2ef46cca82906894d964652. Latest consensus state: current_term: 1 leader_uuid: "c1c7b769b2ef46cca82906894d964652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7b769b2ef46cca82906894d964652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } } }
I20260928 21:11:01.749531 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:01.752389 11263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:01.758351 11263 catalog_manager.cc:1347] Generated new cluster ID: 689aebe3432b4e76a932c3c334f7be5d
I20260928 21:11:01.758438 11263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:01.768326 11263 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:01.768651 11263 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:01.791312 11263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c7b769b2ef46cca82906894d964652: Generated new TSK 0
I20260928 21:11:01.792043 11263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:01.801177 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11176
2026-09-28T21:11:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:01 dist-test-slave-d5wc krb5kdc[11275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:01 dist-test-slave-d5wc krb5kdc[11275](info): set up 2 sockets
Sep 28 21:11:01 dist-test-slave-d5wc krb5kdc[11275](info): commencing operation
krb5kdc: starting...
W20260928 21:11:02.843871 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:02 dist-test-slave-d5wc krb5kdc[11275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:02.968662 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:42365
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:37777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:42365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:03.081276 11288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:03.081614 11288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:03.081691 11288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:03.085795 11288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:03.085906 11288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:03.085947 11288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:03.086006 11288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:03.086045 11288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:03.091549 11288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:37777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:42365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:42365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:03.092656 11288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:03.093677 11288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:03.102667 11293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:03.102660 11294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:03.102665 11296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:03.102743 11288 server_base.cc:1034] running on GCE node
I20260928 21:11:03.103592 11288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:03.104759 11288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:03.105940 11288 hybrid_clock.cc:648] HybridClock initialized: now 1790629863105920 us; error 32 us; skew 500 ppm
Sep 28 21:11:03 dist-test-slave-d5wc krb5kdc[11275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:03.108876 11288 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:03.109841 11288 webserver.cc:460] Webserver started at http://127.21.140.126:41737/ using document root <none> and password file <none>
I20260928 21:11:03.110466 11288 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:03.110569 11288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:03.110787 11288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:03.112850 11288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "e813acb6961f49e3bd3a58dd38a2fcc6"
format_stamp: "Formatted at 2026-09-28 21:11:03 on dist-test-slave-d5wc"
server_key: "16940892b17aaaaad3803010614e7e6e"
server_key_iv: "6c4bf737c7499b02db523667bd8492e4"
server_key_version: "encryptionkey@0"
I20260928 21:11:03.113425 11288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "e813acb6961f49e3bd3a58dd38a2fcc6"
format_stamp: "Formatted at 2026-09-28 21:11:03 on dist-test-slave-d5wc"
server_key: "16940892b17aaaaad3803010614e7e6e"
server_key_iv: "6c4bf737c7499b02db523667bd8492e4"
server_key_version: "encryptionkey@0"
I20260928 21:11:03.117429 11288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 21:11:03.120168 11303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:03.121483 11288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:03.121630 11288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "e813acb6961f49e3bd3a58dd38a2fcc6"
format_stamp: "Formatted at 2026-09-28 21:11:03 on dist-test-slave-d5wc"
server_key: "16940892b17aaaaad3803010614e7e6e"
server_key_iv: "6c4bf737c7499b02db523667bd8492e4"
server_key_version: "encryptionkey@0"
I20260928 21:11:03.121762 11288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:03.142871 11288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:03.146332 11288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:03.146550 11288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:03.154095 11288 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:42365
I20260928 21:11:03.154106 11355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:42365 every 8 connection(s)
I20260928 21:11:03.155083 11288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:03.157799 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:03.163417 11356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6: Bootstrap starting.
I20260928 21:11:03.165410 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11288
I20260928 21:11:03.165586 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:03.165866 22065 external_mini_cluster.cc:1428] Setting key 3cbe22b89b508080f9aa1a3a4b645444
I20260928 21:11:03.166213 11356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:03.167080 11356 log.cc:824] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:03.169445 11356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6: No bootstrap required, opened a new log
I20260928 21:11:03.173945 11356 raft_consensus.cc:348] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42365 } }
I20260928 21:11:03.174249 11356 raft_consensus.cc:374] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:03.174340 11356 raft_consensus.cc:729] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e813acb6961f49e3bd3a58dd38a2fcc6, State: Initialized, Role: FOLLOWER
I20260928 21:11:03.174844 11356 consensus_queue.cc:260] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42365 } }
I20260928 21:11:03.174973 11356 raft_consensus.cc:388] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:03.175050 11356 raft_consensus.cc:482] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:03.175171 11356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:11:03 dist-test-slave-d5wc krb5kdc[11275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:11:03.176537 11356 raft_consensus.cc:504] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42365 } }
I20260928 21:11:03.176918 11356 leader_election.cc:302] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e813acb6961f49e3bd3a58dd38a2fcc6; no voters:
I20260928 21:11:03.177259 11356 leader_election.cc:288] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:03.177556 11361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:03.177865 11361 raft_consensus.cc:686] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [term 1 LEADER]: Becoming Leader. State: Replica: e813acb6961f49e3bd3a58dd38a2fcc6, State: Running, Role: LEADER
I20260928 21:11:03.178285 11361 consensus_queue.cc:237] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42365 } }
I20260928 21:11:03.178655 11356 sys_catalog.cc:564] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:03.180248 11363 sys_catalog.cc:455] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e813acb6961f49e3bd3a58dd38a2fcc6. Latest consensus state: current_term: 1 leader_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42365 } } }
I20260928 21:11:03.180392 11363 sys_catalog.cc:458] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:03.180747 11362 sys_catalog.cc:455] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e813acb6961f49e3bd3a58dd38a2fcc6" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42365 } } }
I20260928 21:11:03.181118 11362 sys_catalog.cc:458] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:03.182832 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:03.167152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47475 (local address 127.21.140.126:42365)
0928 21:11:03.167717 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:03.167732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:03.167768 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:11:03.168680 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:03.168705 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:03.169134 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:03.169520 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:03.170357 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:03.171895 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:03.172770 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:03.173497 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:03.173680 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:03.176770 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:03.176776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:03.176790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:03.176821 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:03.179061 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:03.179770 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:03.179777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:03.179784 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:03.179938 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:03.180337 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:03.180341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:03.180342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:03.180933 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:03.181138 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:03.181590 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 21:11:03.181829 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":148,"threads_started":1}
W20260928 21:11:03.183774 11376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:03.183856 11376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:03.183960 11375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:03.184778 11375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:03.186167 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:03.190932 11375 catalog_manager.cc:1347] Generated new cluster ID: 0939e0f759924f1e851370d5842cc7cd
I20260928 21:11:03.190999 11375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:03.198799 11375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:03.199414 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:03.188620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54031 (local address 127.21.140.126:42365)
0928 21:11:03.188739 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:03.188746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:03.190288 (+ 1542us) server_negotiation.cc:407] Connection header received
0928 21:11:03.190454 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:03.190457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:03.190497 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:03.190590 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:03.191555 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:03.192372 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:03.193498 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:03.193855 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:03.193994 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:03.195624 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:03.195628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:03.195631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:03.195660 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:03.197404 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:03.198017 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:03.198021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:03.198023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:03.198112 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:03.198520 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:03.198523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:03.198525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:03.198713 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:03.198873 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:03.199148 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:11:03.199249 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:11:03.200275 11375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:03.208878 11375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e813acb6961f49e3bd3a58dd38a2fcc6: Generated new TSK 0
I20260928 21:11:03.209816 11375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:03.213217 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11288
W20260928 21:11:03.218832 11273 connection.cc:535] client connection to 127.21.140.126:42365 recv error: Network error: failed to read from TLS socket (remote: 127.21.140.126:42365): Connection reset by peer (error 104)
2026-09-28T21:11:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:03 dist-test-slave-d5wc krb5kdc[11387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:03 dist-test-slave-d5wc krb5kdc[11387](info): set up 2 sockets
Sep 28 21:11:03 dist-test-slave-d5wc krb5kdc[11387](info): commencing operation
krb5kdc: starting...
W20260928 21:11:04.243808 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:04 dist-test-slave-d5wc krb5kdc[11387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:04.367174 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40241
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:04.479972 11400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:04.480290 11400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:04.480334 11400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:04.484527 11400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:04.484606 11400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:04.484627 11400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:04.484643 11400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:04.484663 11400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:04.490232 11400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:40241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:04.491329 11400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:04.492307 11400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:04.500943 11408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:04.500943 11405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:04.500958 11400 server_base.cc:1034] running on GCE node
W20260928 21:11:04.500937 11406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:04.501797 11400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:04.502959 11400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:04.504124 11400 hybrid_clock.cc:648] HybridClock initialized: now 1790629864504107 us; error 32 us; skew 500 ppm
Sep 28 21:11:04 dist-test-slave-d5wc krb5kdc[11387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629864, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:04.512418 11400 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:04.513552 11400 webserver.cc:460] Webserver started at http://127.21.140.126:46865/ using document root <none> and password file <none>
I20260928 21:11:04.514216 11400 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:04.514286 11400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:04.514528 11400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:04.516659 11400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "dec87b2bea864a1d9a5da3a2f105b489"
format_stamp: "Formatted at 2026-09-28 21:11:04 on dist-test-slave-d5wc"
server_key: "83bdad7e611c7282a6bf9825af917c6a"
server_key_iv: "446a8c5f5dde898eda9b1add556a80ac"
server_key_version: "encryptionkey@0"
I20260928 21:11:04.517258 11400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "dec87b2bea864a1d9a5da3a2f105b489"
format_stamp: "Formatted at 2026-09-28 21:11:04 on dist-test-slave-d5wc"
server_key: "83bdad7e611c7282a6bf9825af917c6a"
server_key_iv: "446a8c5f5dde898eda9b1add556a80ac"
server_key_version: "encryptionkey@0"
I20260928 21:11:04.521163 11400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:11:04.523741 11415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:04.525035 11400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:04.525187 11400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "dec87b2bea864a1d9a5da3a2f105b489"
format_stamp: "Formatted at 2026-09-28 21:11:04 on dist-test-slave-d5wc"
server_key: "83bdad7e611c7282a6bf9825af917c6a"
server_key_iv: "446a8c5f5dde898eda9b1add556a80ac"
server_key_version: "encryptionkey@0"
I20260928 21:11:04.525309 11400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:04.545027 11400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:04.548731 11400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:04.548952 11400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:04.556908 11400 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:40241
I20260928 21:11:04.556941 11467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:40241 every 8 connection(s)
I20260928 21:11:04.557883 11400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:04.560752 11468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:04.564011 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11400
I20260928 21:11:04.564167 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:04.564456 22065 external_mini_cluster.cc:1428] Setting key a99787544b3658a88c95b20f85bb5640
I20260928 21:11:04.566998 11468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489: Bootstrap starting.
I20260928 21:11:04.569180 11468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:04.569895 11468 log.cc:824] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:04.572028 11468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489: No bootstrap required, opened a new log
I20260928 21:11:04.574379 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:04.566019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54995 (local address 127.21.140.126:40241)
0928 21:11:04.566461 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:04.566476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:04.566513 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:11:04.567249 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:04.567267 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:04.567611 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:04.567957 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:04.568371 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:04.569421 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:04.570429 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:04.571023 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:04.571223 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:04.572106 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:04.572112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:04.572129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:04.572626 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:04.572692 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:04.572721 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:04.572857 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:04.573029 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:11:04.573167 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":130,"threads_started":1}
I20260928 21:11:04.575968 11468 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec87b2bea864a1d9a5da3a2f105b489" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40241 } }
I20260928 21:11:04.576237 11468 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:04.576328 11468 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec87b2bea864a1d9a5da3a2f105b489, State: Initialized, Role: FOLLOWER
I20260928 21:11:04.576821 11468 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec87b2bea864a1d9a5da3a2f105b489" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40241 } }
I20260928 21:11:04.576985 11468 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:04.577072 11468 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:04.577174 11468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:04.577395 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:04.578214 11468 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec87b2bea864a1d9a5da3a2f105b489" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40241 } }
I20260928 21:11:04.578564 11468 leader_election.cc:302] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec87b2bea864a1d9a5da3a2f105b489; no voters:
I20260928 21:11:04.578925 11468 leader_election.cc:288] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:04.578989 11477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:04.579214 11477 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [term 1 LEADER]: Becoming Leader. State: Replica: dec87b2bea864a1d9a5da3a2f105b489, State: Running, Role: LEADER
I20260928 21:11:04.579564 11477 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec87b2bea864a1d9a5da3a2f105b489" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40241 } }
I20260928 21:11:04.580230 11468 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:04.581418 11478 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec87b2bea864a1d9a5da3a2f105b489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec87b2bea864a1d9a5da3a2f105b489" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40241 } } }
I20260928 21:11:04.581547 11478 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:04.581894 11479 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec87b2bea864a1d9a5da3a2f105b489. Latest consensus state: current_term: 1 leader_uuid: "dec87b2bea864a1d9a5da3a2f105b489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec87b2bea864a1d9a5da3a2f105b489" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 40241 } } }
I20260928 21:11:04.582103 11479 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:04.582731 11487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:04.584393 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:04.579815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56861 (local address 127.21.140.126:40241)
0928 21:11:04.579915 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:04.579920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:04.580000 (+ 80us) server_negotiation.cc:407] Connection header received
0928 21:11:04.580138 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:04.580144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:04.580205 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:04.580281 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:04.580558 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:04.581299 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:04.582451 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:04.582787 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:04.582889 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:04.583918 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:04.583922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:04.583925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:04.583937 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:04.583972 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:04.583983 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:04.584084 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:04.584208 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:11:04.584241 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:11:04.585831 11487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:04.591567 11487 catalog_manager.cc:1347] Generated new cluster ID: 2d806041051744e7badb3de884ac3a3a
I20260928 21:11:04.591667 11487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:04.602826 11487 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:04.603883 11487 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:04.610334 11487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec87b2bea864a1d9a5da3a2f105b489: Generated new TSK 0
I20260928 21:11:04.611048 11487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:04.619506 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11400
W20260928 21:11:04.624852 11385 connection.cc:535] client connection to 127.21.140.126:40241 recv error: Network error: failed to read from TLS socket (remote: 127.21.140.126:40241): Connection reset by peer (error 104)
2026-09-28T21:11:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:04 dist-test-slave-d5wc krb5kdc[11499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:04 dist-test-slave-d5wc krb5kdc[11499](info): set up 2 sockets
Sep 28 21:11:04 dist-test-slave-d5wc krb5kdc[11499](info): commencing operation
krb5kdc: starting...
W20260928 21:11:05.683645 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:05 dist-test-slave-d5wc krb5kdc[11499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:05.807133 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:45955
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:05.917800 11512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:05.918165 11512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:05.918256 11512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:05.922304 11512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:05.922406 11512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:05.922446 11512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:05.922513 11512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:05.922554 11512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:05.927805 11512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:45955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:05.928886 11512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:05.929899 11512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:11:05.938669 11512 server_base.cc:1034] running on GCE node
W20260928 21:11:05.938607 11517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:05.938608 11520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:05.938608 11518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:05.939594 11512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:05.940632 11512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:05.941798 11512 hybrid_clock.cc:648] HybridClock initialized: now 1790629865941779 us; error 31 us; skew 500 ppm
Sep 28 21:11:05 dist-test-slave-d5wc krb5kdc[11499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629865, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:05.944641 11512 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:05.945510 11512 webserver.cc:460] Webserver started at http://127.21.140.126:44707/ using document root <none> and password file <none>
I20260928 21:11:05.946137 11512 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:05.946249 11512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:05.946482 11512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:05.948515 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "489edc807cf44a84afef756e801abd5f"
format_stamp: "Formatted at 2026-09-28 21:11:05 on dist-test-slave-d5wc"
server_key: "e02d5cc6a7ac3d8f3b9e8f9a8d37376e"
server_key_iv: "9cfede9009d395f2a45f95e39442397a"
server_key_version: "encryptionkey@0"
I20260928 21:11:05.949081 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "489edc807cf44a84afef756e801abd5f"
format_stamp: "Formatted at 2026-09-28 21:11:05 on dist-test-slave-d5wc"
server_key: "e02d5cc6a7ac3d8f3b9e8f9a8d37376e"
server_key_iv: "9cfede9009d395f2a45f95e39442397a"
server_key_version: "encryptionkey@0"
I20260928 21:11:05.953157 11512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 21:11:05.956403 11527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:05.957662 11512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:11:05.957810 11512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "489edc807cf44a84afef756e801abd5f"
format_stamp: "Formatted at 2026-09-28 21:11:05 on dist-test-slave-d5wc"
server_key: "e02d5cc6a7ac3d8f3b9e8f9a8d37376e"
server_key_iv: "9cfede9009d395f2a45f95e39442397a"
server_key_version: "encryptionkey@0"
I20260928 21:11:05.957930 11512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:05.962826 11512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:05.966233 11512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:05.966434 11512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:05.973610 11512 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:45955
I20260928 21:11:05.973627 11579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:45955 every 8 connection(s)
I20260928 21:11:05.974622 11512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:05.977293 11580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:05.982842 11580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f: Bootstrap starting.
I20260928 21:11:05.983206 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11512
I20260928 21:11:05.983362 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:05.983619 22065 external_mini_cluster.cc:1428] Setting key ca0776ec8d8617a511b4a5b0a71d1d44
I20260928 21:11:05.985585 11580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:05.986496 11580 log.cc:824] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:05.988567 11580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f: No bootstrap required, opened a new log
I20260928 21:11:05.989218 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:05.985010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48617 (local address 127.21.140.126:45955)
0928 21:11:05.985464 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:05.985485 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:05.985530 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:11:05.986471 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:05.986495 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:05.986864 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:05.987236 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:05.988019 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:05.988023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:05.988035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:05.988045 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:05.988101 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:05.988120 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:05.988238 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:05.988340 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:11:05.988498 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260928 21:11:05.991914 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:05.993214 11580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489edc807cf44a84afef756e801abd5f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45955 } }
I20260928 21:11:05.993461 11580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:05.993538 11580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489edc807cf44a84afef756e801abd5f, State: Initialized, Role: FOLLOWER
I20260928 21:11:05.994083 11580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489edc807cf44a84afef756e801abd5f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45955 } }
I20260928 21:11:05.994262 11580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:05.994331 11580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:05.994444 11580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:05.995603 11580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489edc807cf44a84afef756e801abd5f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45955 } }
I20260928 21:11:05.995940 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:05.994028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37847 (local address 127.21.140.126:45955)
0928 21:11:05.994138 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:05.994143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:05.994371 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:11:05.994583 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:05.994586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:05.994626 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:05.994714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:05.995521 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:05.995525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:05.995527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:05.995539 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:05.995569 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:05.995580 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:05.995674 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:05.995760 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:11:05.995799 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:11:05.996024 11580 leader_election.cc:302] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 489edc807cf44a84afef756e801abd5f; no voters:
I20260928 21:11:05.996387 11580 leader_election.cc:288] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:05.996476 11590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:05.996665 11590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [term 1 LEADER]: Becoming Leader. State: Replica: 489edc807cf44a84afef756e801abd5f, State: Running, Role: LEADER
I20260928 21:11:05.997237 11590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489edc807cf44a84afef756e801abd5f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45955 } }
I20260928 21:11:05.997747 11580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:05.999195 11591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "489edc807cf44a84afef756e801abd5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489edc807cf44a84afef756e801abd5f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45955 } } }
I20260928 21:11:05.999178 11592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 489edc807cf44a84afef756e801abd5f. Latest consensus state: current_term: 1 leader_uuid: "489edc807cf44a84afef756e801abd5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "489edc807cf44a84afef756e801abd5f" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 45955 } } }
I20260928 21:11:05.999364 11591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:05.999363 11592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:05.999758 11601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:06.002645 11601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:06.007843 11601 catalog_manager.cc:1347] Generated new cluster ID: fb03e49d4144496bbb14028169527c25
I20260928 21:11:06.007977 11601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:06.014415 11601 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:06.014763 11601 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:06.018925 11601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 489edc807cf44a84afef756e801abd5f: Generated new TSK 0
I20260928 21:11:06.019693 11601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:06.029349 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11512
2026-09-28T21:11:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:06 dist-test-slave-d5wc krb5kdc[11611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:06 dist-test-slave-d5wc krb5kdc[11611](info): set up 2 sockets
Sep 28 21:11:06 dist-test-slave-d5wc krb5kdc[11611](info): commencing operation
krb5kdc: starting...
W20260928 21:11:07.073053 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:07 dist-test-slave-d5wc krb5kdc[11611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:07.221397 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37433
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:37433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:07.357764 11624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:07.358278 11624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:07.358383 11624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:07.363034 11624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:07.363189 11624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:07.363237 11624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:07.363279 11624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:07.363314 11624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:07.369717 11624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:37433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:37433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:07.371345 11624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:07.372772 11624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:07.385531 11629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:07.385531 11630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:07.385738 11624 server_base.cc:1034] running on GCE node
W20260928 21:11:07.385826 11632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:07.387108 11624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:07.388823 11624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:07.390082 11624 hybrid_clock.cc:648] HybridClock initialized: now 1790629867390060 us; error 47 us; skew 500 ppm
Sep 28 21:11:07 dist-test-slave-d5wc krb5kdc[11611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:07.394029 11624 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:07.395401 11624 webserver.cc:460] Webserver started at http://127.21.140.126:39207/ using document root <none> and password file <none>
I20260928 21:11:07.396196 11624 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:07.396308 11624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:07.396616 11624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:07.399581 11624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "528001cf8a704cedabe455cbe0bb5fce"
format_stamp: "Formatted at 2026-09-28 21:11:07 on dist-test-slave-d5wc"
server_key: "aa9693c394b029bf4a3c71871131d187"
server_key_iv: "46441680d8ab63d0b8ce710fa704dfb0"
server_key_version: "encryptionkey@0"
I20260928 21:11:07.400395 11624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "528001cf8a704cedabe455cbe0bb5fce"
format_stamp: "Formatted at 2026-09-28 21:11:07 on dist-test-slave-d5wc"
server_key: "aa9693c394b029bf4a3c71871131d187"
server_key_iv: "46441680d8ab63d0b8ce710fa704dfb0"
server_key_version: "encryptionkey@0"
I20260928 21:11:07.405802 11624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 21:11:07.409401 11639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:07.411463 11624 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:11:07.411696 11624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "528001cf8a704cedabe455cbe0bb5fce"
format_stamp: "Formatted at 2026-09-28 21:11:07 on dist-test-slave-d5wc"
server_key: "aa9693c394b029bf4a3c71871131d187"
server_key_iv: "46441680d8ab63d0b8ce710fa704dfb0"
server_key_version: "encryptionkey@0"
I20260928 21:11:07.411881 11624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:07.424822 11624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:07.429291 11624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:07.429605 11624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:07.439205 11624 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:37433
I20260928 21:11:07.439227 11691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:37433 every 8 connection(s)
I20260928 21:11:07.440662 11624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:07.444782 11692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:07.450932 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11624
I20260928 21:11:07.451107 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:07.451438 22065 external_mini_cluster.cc:1428] Setting key 80bcb9e9be9a039560165bad3b1bfbad
I20260928 21:11:07.453469 11692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce: Bootstrap starting.
I20260928 21:11:07.456734 11692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:07.457765 11692 log.cc:824] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:07.460505 11692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce: No bootstrap required, opened a new log
Sep 28 21:11:07 dist-test-slave-d5wc krb5kdc[11611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:11:07.465528 11692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528001cf8a704cedabe455cbe0bb5fce" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37433 } }
I20260928 21:11:07.465890 11692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:07.466012 11692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528001cf8a704cedabe455cbe0bb5fce, State: Initialized, Role: FOLLOWER
I20260928 21:11:07.466711 11692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528001cf8a704cedabe455cbe0bb5fce" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37433 } }
I20260928 21:11:07.466930 11692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:07.467046 11692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:07.467185 11692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:07.468777 11692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528001cf8a704cedabe455cbe0bb5fce" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37433 } }
I20260928 21:11:07.469304 11692 leader_election.cc:302] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528001cf8a704cedabe455cbe0bb5fce; no voters:
I20260928 21:11:07.469789 11692 leader_election.cc:288] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:07.469954 11697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:07.470264 11697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [term 1 LEADER]: Becoming Leader. State: Replica: 528001cf8a704cedabe455cbe0bb5fce, State: Running, Role: LEADER
I20260928 21:11:07.470399 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:07.453425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36737 (local address 127.21.140.126:37433)
0928 21:11:07.454094 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:07.454115 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:07.454164 (+ 49us) server_negotiation.cc:407] Connection header received
0928 21:11:07.455018 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:07.455043 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:07.455634 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:07.456249 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:07.457335 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:07.458786 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:07.459815 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:07.460393 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:07.460623 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:07.463650 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:07.463657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:07.463678 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:07.463725 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:07.465957 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:07.466665 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:07.466671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:07.466677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:07.466876 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:07.467312 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:07.467316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:07.467318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:07.467912 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:07.468136 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:07.468527 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 21:11:07.469022 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":189,"threads_started":1}
I20260928 21:11:07.470788 11697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528001cf8a704cedabe455cbe0bb5fce" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37433 } }
I20260928 21:11:07.471642 11692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:07.473465 11698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528001cf8a704cedabe455cbe0bb5fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528001cf8a704cedabe455cbe0bb5fce" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37433 } } }
I20260928 21:11:07.473661 11698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:07.474028 11699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528001cf8a704cedabe455cbe0bb5fce. Latest consensus state: current_term: 1 leader_uuid: "528001cf8a704cedabe455cbe0bb5fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528001cf8a704cedabe455cbe0bb5fce" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 37433 } } }
I20260928 21:11:07.474161 11699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:07.474967 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 21:11:07.478988 11712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:07.479176 11712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:07.479288 11707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:07.480170 11707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:07.488517 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:07.477835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36393 (local address 127.21.140.126:37433)
0928 21:11:07.478035 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:07.478042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:07.478346 (+ 304us) server_negotiation.cc:407] Connection header received
0928 21:11:07.478568 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:07.478571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:07.478628 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:07.478772 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:07.479772 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:07.480472 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:07.481433 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:07.481724 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:07.481861 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:07.483997 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:07.484002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:07.484006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:07.484045 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:07.486183 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:07.486830 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:07.486835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:07.486837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:07.486959 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:07.487327 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:07.487330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:07.487333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:07.487627 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:07.487797 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:07.488115 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 21:11:07.488290 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:11:07.489033 11707 catalog_manager.cc:1347] Generated new cluster ID: 97daaa7586bc457a892bab1545105efd
I20260928 21:11:07.489114 11707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:07.512182 11707 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:07.513497 11707 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:07.519131 11707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528001cf8a704cedabe455cbe0bb5fce: Generated new TSK 0
I20260928 21:11:07.520174 11707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:07.533138 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11624
2026-09-28T21:11:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:07 dist-test-slave-d5wc krb5kdc[11723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:07 dist-test-slave-d5wc krb5kdc[11723](info): set up 2 sockets
Sep 28 21:11:07 dist-test-slave-d5wc krb5kdc[11723](info): commencing operation
krb5kdc: starting...
W20260928 21:11:08.632323 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:08 dist-test-slave-d5wc krb5kdc[11723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:08.757426 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:39787
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:08.879503 11736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:08.879842 11736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:08.879944 11736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:08.884187 11736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:08.884292 11736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:08.884331 11736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:08.884393 11736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:08.884438 11736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:08.889985 11736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:39787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:08.891247 11736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:08.892416 11736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:08.901786 11741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:08.901783 11742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:08.902082 11736 server_base.cc:1034] running on GCE node
W20260928 21:11:08.901777 11744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:08.902837 11736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:08.904222 11736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:08.905445 11736 hybrid_clock.cc:648] HybridClock initialized: now 1790629868905423 us; error 37 us; skew 500 ppm
Sep 28 21:11:08 dist-test-slave-d5wc krb5kdc[11723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629868, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:08.908696 11736 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:08.909674 11736 webserver.cc:460] Webserver started at http://127.21.140.126:39559/ using document root <none> and password file <none>
I20260928 21:11:08.910362 11736 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:08.910463 11736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:08.910718 11736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:08.912931 11736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "3cba84fcb06a46318e3a6aa4c3d0b470"
format_stamp: "Formatted at 2026-09-28 21:11:08 on dist-test-slave-d5wc"
server_key: "b255e6e9997134b0adf7771be5ae1bab"
server_key_iv: "87b8ca5479751e91b82ce4655b5506c9"
server_key_version: "encryptionkey@0"
I20260928 21:11:08.913524 11736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "3cba84fcb06a46318e3a6aa4c3d0b470"
format_stamp: "Formatted at 2026-09-28 21:11:08 on dist-test-slave-d5wc"
server_key: "b255e6e9997134b0adf7771be5ae1bab"
server_key_iv: "87b8ca5479751e91b82ce4655b5506c9"
server_key_version: "encryptionkey@0"
I20260928 21:11:08.917551 11736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:11:08.920264 11751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:08.921634 11736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:11:08.921798 11736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "3cba84fcb06a46318e3a6aa4c3d0b470"
format_stamp: "Formatted at 2026-09-28 21:11:08 on dist-test-slave-d5wc"
server_key: "b255e6e9997134b0adf7771be5ae1bab"
server_key_iv: "87b8ca5479751e91b82ce4655b5506c9"
server_key_version: "encryptionkey@0"
I20260928 21:11:08.921922 11736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:08.946559 11736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:08.949935 11736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:08.950217 11736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:08.957477 11736 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:39787
I20260928 21:11:08.957755 11803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:39787 every 8 connection(s)
I20260928 21:11:08.958600 11736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:08.961712 11804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:08.964874 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11736
I20260928 21:11:08.965016 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:08.965340 22065 external_mini_cluster.cc:1428] Setting key 987fccc3b35b1e9a87dd5d31cf843181
I20260928 21:11:08.968408 11804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470: Bootstrap starting.
I20260928 21:11:08.971342 11804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:08.972252 11804 log.cc:824] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:08.974785 11804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470: No bootstrap required, opened a new log
I20260928 21:11:08.976913 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:08.967367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49607 (local address 127.21.140.126:39787)
0928 21:11:08.968238 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:08.968258 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:08.968302 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:11:08.969158 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:08.969183 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:08.969658 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:08.970111 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:08.970539 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:08.972026 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:08.973211 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:08.973905 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:08.974101 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:08.975033 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:08.975039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:08.975052 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:08.975437 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:08.975491 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:08.975517 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:08.975635 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:08.975765 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:11:08.975970 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":127,"threads_started":1}
I20260928 21:11:08.978086 11804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39787 } }
I20260928 21:11:08.978363 11804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:08.978418 11804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cba84fcb06a46318e3a6aa4c3d0b470, State: Initialized, Role: FOLLOWER
I20260928 21:11:08.978879 11804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39787 } }
I20260928 21:11:08.979074 11804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:08.979120 11804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:08.979221 11804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:08.980039 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:08.980435 11804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39787 } }
I20260928 21:11:08.980791 11804 leader_election.cc:302] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cba84fcb06a46318e3a6aa4c3d0b470; no voters:
I20260928 21:11:08.981149 11804 leader_election.cc:288] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:08.981215 11809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:08.981432 11809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [term 1 LEADER]: Becoming Leader. State: Replica: 3cba84fcb06a46318e3a6aa4c3d0b470, State: Running, Role: LEADER
I20260928 21:11:08.981755 11809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39787 } }
I20260928 21:11:08.982517 11804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:08.983001 11811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cba84fcb06a46318e3a6aa4c3d0b470. Latest consensus state: current_term: 1 leader_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39787 } } }
I20260928 21:11:08.983227 11811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:08.983078 11812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cba84fcb06a46318e3a6aa4c3d0b470" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 39787 } } }
I20260928 21:11:08.983348 11812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:08.983788 11823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:08.987229 11823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:08.988092 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:08.982831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47609 (local address 127.21.140.126:39787)
0928 21:11:08.982944 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:08.982949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:08.983685 (+ 736us) server_negotiation.cc:407] Connection header received
0928 21:11:08.983879 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:08.983883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:08.983943 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:08.983994 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:08.984411 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:08.985150 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:08.986289 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:08.986630 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:08.986743 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:08.987636 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:08.987640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:08.987643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:08.987656 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:08.987684 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:08.987698 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:08.987764 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:08.987902 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:11:08.987969 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:11:08.994187 11823 catalog_manager.cc:1347] Generated new cluster ID: 838fb8392a414c84b4866c70c8eac330
I20260928 21:11:08.994282 11823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:09.009822 11823 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:09.010934 11823 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:09.018517 11823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cba84fcb06a46318e3a6aa4c3d0b470: Generated new TSK 0
I20260928 21:11:09.019336 11823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:09.025265 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11736
2026-09-28T21:11:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:09 dist-test-slave-d5wc krb5kdc[11835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:09 dist-test-slave-d5wc krb5kdc[11835](info): set up 2 sockets
Sep 28 21:11:09 dist-test-slave-d5wc krb5kdc[11835](info): commencing operation
krb5kdc: starting...
W20260928 21:11:10.151819 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:10 dist-test-slave-d5wc krb5kdc[11835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:10.276942 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:42673
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:42673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:10.388403 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:10.388774 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:10.388857 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:10.393131 11848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:10.393244 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:10.393280 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:10.393321 11848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:10.393353 11848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:10.398995 11848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:42673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:42673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:10.400157 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:10.401247 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:10.410293 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:10.410293 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:10.410336 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:10.410545 11848 server_base.cc:1034] running on GCE node
I20260928 21:11:10.411249 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:10.412433 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:10.413717 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790629870413669 us; error 43 us; skew 500 ppm
Sep 28 21:11:10 dist-test-slave-d5wc krb5kdc[11835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629870, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:10.423858 11848 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:10.424980 11848 webserver.cc:460] Webserver started at http://127.21.140.126:37321/ using document root <none> and password file <none>
I20260928 21:11:10.425606 11848 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:10.425704 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:10.425956 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:10.428189 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "9af2fadc7689400d9f398d0804a26221"
format_stamp: "Formatted at 2026-09-28 21:11:10 on dist-test-slave-d5wc"
server_key: "09d8837bfb631cbd7eb2d8d1b8faa6e7"
server_key_iv: "d8ec2e56d63c01336bed92acf0c3d63a"
server_key_version: "encryptionkey@0"
I20260928 21:11:10.428815 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "9af2fadc7689400d9f398d0804a26221"
format_stamp: "Formatted at 2026-09-28 21:11:10 on dist-test-slave-d5wc"
server_key: "09d8837bfb631cbd7eb2d8d1b8faa6e7"
server_key_iv: "d8ec2e56d63c01336bed92acf0c3d63a"
server_key_version: "encryptionkey@0"
I20260928 21:11:10.432811 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 21:11:10.435498 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:10.436784 11848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:11:10.436937 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "9af2fadc7689400d9f398d0804a26221"
format_stamp: "Formatted at 2026-09-28 21:11:10 on dist-test-slave-d5wc"
server_key: "09d8837bfb631cbd7eb2d8d1b8faa6e7"
server_key_iv: "d8ec2e56d63c01336bed92acf0c3d63a"
server_key_version: "encryptionkey@0"
I20260928 21:11:10.437067 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:10.441874 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:10.445115 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:10.445330 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:10.452559 11848 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:42673
I20260928 21:11:10.452579 11915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:42673 every 8 connection(s)
I20260928 21:11:10.453518 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:10.456257 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:10.461886 11916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221: Bootstrap starting.
I20260928 21:11:10.463721 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11848
I20260928 21:11:10.463857 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:10.464133 22065 external_mini_cluster.cc:1428] Setting key 23f2a951d14936975498f2fb92d08ccd
I20260928 21:11:10.464782 11916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:10.465699 11916 log.cc:824] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:10.467584 11916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221: No bootstrap required, opened a new log
I20260928 21:11:10.469462 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:10.465705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39281 (local address 127.21.140.126:42673)
0928 21:11:10.466195 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:10.466211 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:10.466246 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:11:10.466888 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:10.466907 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:10.467189 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:10.467547 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:10.468282 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:10.468287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:10.468298 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:10.468308 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:10.468358 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:10.468378 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:10.468508 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:10.468610 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:11:10.468764 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":183,"threads_started":1}
I20260928 21:11:10.471438 11916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af2fadc7689400d9f398d0804a26221" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42673 } }
I20260928 21:11:10.471748 11916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:10.471829 11916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af2fadc7689400d9f398d0804a26221, State: Initialized, Role: FOLLOWER
I20260928 21:11:10.472339 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:10.472285 11916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af2fadc7689400d9f398d0804a26221" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42673 } }
I20260928 21:11:10.472447 11916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:10.472538 11916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:10.472636 11916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:10.473626 11916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af2fadc7689400d9f398d0804a26221" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42673 } }
I20260928 21:11:10.474041 11916 leader_election.cc:302] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af2fadc7689400d9f398d0804a26221; no voters:
I20260928 21:11:10.474400 11916 leader_election.cc:288] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:10.474457 11925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:10.474679 11925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [term 1 LEADER]: Becoming Leader. State: Replica: 9af2fadc7689400d9f398d0804a26221, State: Running, Role: LEADER
I20260928 21:11:10.474977 11925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af2fadc7689400d9f398d0804a26221" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42673 } }
I20260928 21:11:10.475696 11916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:10.476141 11925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9af2fadc7689400d9f398d0804a26221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af2fadc7689400d9f398d0804a26221" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42673 } } }
I20260928 21:11:10.476253 11925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:10.476646 11930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:10.477022 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:10.475153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37159 (local address 127.21.140.126:42673)
0928 21:11:10.475287 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:10.475294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:10.475325 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:11:10.475469 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:10.475474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:10.475516 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:10.475621 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:10.476379 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:10.476383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:10.476386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:11:10.476397 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:10.476425 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:11:10.476438 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:10.476514 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:10.476857 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 21:11:10.476895 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:11:10.478256 11926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af2fadc7689400d9f398d0804a26221. Latest consensus state: current_term: 1 leader_uuid: "9af2fadc7689400d9f398d0804a26221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af2fadc7689400d9f398d0804a26221" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 42673 } } }
I20260928 21:11:10.478417 11926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:10.480572 11930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:10.486536 11930 catalog_manager.cc:1347] Generated new cluster ID: b74fce176f5d4c24b0071ef126101ae2
I20260928 21:11:10.486637 11930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:10.506497 11930 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:10.506844 11930 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:10.513712 11930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9af2fadc7689400d9f398d0804a26221: Generated new TSK 0
I20260928 21:11:10.514469 11930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:11:10.520967 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11848
2026-09-28T21:11:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1495 ms)
[----------] 18 tests from AuthTokenIssuingTest (25864 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:10 dist-test-slave-d5wc krb5kdc[11947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:10 dist-test-slave-d5wc krb5kdc[11947](info): set up 2 sockets
Sep 28 21:11:10 dist-test-slave-d5wc krb5kdc[11947](info): commencing operation
krb5kdc: starting...
W20260928 21:11:11.567844 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:11 dist-test-slave-d5wc krb5kdc[11947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:11.694648 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44967
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44967,127.21.140.125:39473,127.21.140.124:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:11.805140 11960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:11:11.805522 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:11.805610 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:11.805691 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:11.809721 11960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:11.809828 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:11.809865 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:11.809901 11960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:11.809926 11960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:11.815248 11960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44967,127.21.140.125:39473,127.21.140.124:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:11.816356 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:11.817422 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:11.826306 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:11.826309 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:11.826363 11960 server_base.cc:1034] running on GCE node
W20260928 21:11:11.826323 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:11.827179 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:11.828285 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:11.829527 11960 hybrid_clock.cc:648] HybridClock initialized: now 1790629871829514 us; error 26 us; skew 500 ppm
Sep 28 21:11:11 dist-test-slave-d5wc krb5kdc[11947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:11.832511 11960 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:11.833500 11960 webserver.cc:460] Webserver started at http://127.21.140.126:43011/ using document root <none> and password file <none>
I20260928 21:11:11.834137 11960 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:11.834206 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:11.834420 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:11.836479 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "b9373cf3df5b4b7cb66f3109ad893b26"
format_stamp: "Formatted at 2026-09-28 21:11:11 on dist-test-slave-d5wc"
server_key: "ab36fbb602a27daf2bb15df860851d52"
server_key_iv: "dd47c614aaa4d1e9b51bac64cb3b5112"
server_key_version: "encryptionkey@0"
I20260928 21:11:11.837064 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "b9373cf3df5b4b7cb66f3109ad893b26"
format_stamp: "Formatted at 2026-09-28 21:11:11 on dist-test-slave-d5wc"
server_key: "ab36fbb602a27daf2bb15df860851d52"
server_key_iv: "dd47c614aaa4d1e9b51bac64cb3b5112"
server_key_version: "encryptionkey@0"
I20260928 21:11:11.840900 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:11:11.843377 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:11.844626 11960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:11:11.844767 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "b9373cf3df5b4b7cb66f3109ad893b26"
format_stamp: "Formatted at 2026-09-28 21:11:11 on dist-test-slave-d5wc"
server_key: "ab36fbb602a27daf2bb15df860851d52"
server_key_iv: "dd47c614aaa4d1e9b51bac64cb3b5112"
server_key_version: "encryptionkey@0"
I20260928 21:11:11.844882 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:11.872591 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:11.876199 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:11.876433 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:11.884167 11960 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:44967
I20260928 21:11:11.884459 12027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:44967 every 8 connection(s)
I20260928 21:11:11.885254 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:11.888088 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:11.891156 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 11960
I20260928 21:11:11.891319 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:11.891584 22065 external_mini_cluster.cc:1428] Setting key 811cd19c28885785019b77d24aaf3778
I20260928 21:11:11.891970 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.140.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.125@KRBTEST.COM" created.
Sep 28 21:11:11 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:11:11.908594 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:11.895181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.21.140.126:44967)
0928 21:11:11.895484 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:11.895494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:11.895538 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:11:11.896192 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:11.896196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:11.896372 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:11.896573 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:11.897475 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:11.898401 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:11.899381 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:11.899790 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:11.899967 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:11.903363 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:11.903369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:11.903379 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:11.903399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:11.905155 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:11.905665 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:11.905677 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:11.905678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:11.905777 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:11.906164 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:11.906169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:11.906171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:11.906584 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:11.906730 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:11.907151 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 21:11:11.907482 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260928 21:11:11.908594 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:11.894226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44967 (local address 127.0.0.1:47308)
0928 21:11:11.894795 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:11.894841 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:11.895871 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:11.896570 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:11.896574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:11.897056 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:11.897297 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:11.897312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:11.898533 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:11.898538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:11.899254 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:11.899263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:11.899966 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:11.899970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:11.900085 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:11.900842 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:11.900859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:11.903196 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:11.905292 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:11.905300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:11.905311 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:11.905543 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:11.905876 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:11.905879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:11.905881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:11.906063 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:11.906700 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:11.906707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:11.906885 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 21:11:11.907068 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 21:11:11.907533 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":123,"thread_start_us":147,"threads_started":1}
I20260928 21:11:11.911221 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:11.912075 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:11.911755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46688)
0928 21:11:11.911932 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:11.912007 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 21:11:11.912284 11977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
I20260928 21:11:11.913242 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:11.913012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46692)
0928 21:11:11.913137 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:11.913189 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 21:11:11.913496 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39473: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab.
Entry for principal kudu/127.21.140.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab.
I20260928 21:11:11.917639 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:39473
--webserver_interface=127.21.140.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab
--principal=kudu/127.21.140.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44967,127.21.140.125:39473,127.21.140.124:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:11:11.962769 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } attempt: 1
I20260928 21:11:11.963841 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:11.963396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46708)
0928 21:11:11.963655 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:11.963733 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 21:11:11.965451 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:11.964869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46718)
0928 21:11:11.965296 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:11.965359 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
W20260928 21:11:11.965655 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39473: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
I20260928 21:11:12.033819 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } attempt: 2
I20260928 21:11:12.034750 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.034404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46730)
0928 21:11:12.034585 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.034658 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 21:11:12.035553 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.035309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46746)
0928 21:11:12.035434 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.035487 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 21:11:12.035748 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39473: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
W20260928 21:11:12.046511 12035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:11:12.046806 12035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:12.046871 12035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:12.046949 12035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:12.051263 12035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:12.051332 12035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:12.051353 12035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:12.051370 12035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:12.051425 12035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:12.056963 12035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44967,127.21.140.125:39473,127.21.140.124:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab
--principal=kudu/127.21.140.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.140.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:12.058177 12035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:12.059233 12035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:12.068241 12043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:12.068399 12035 server_base.cc:1034] running on GCE node
W20260928 21:11:12.068233 12041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:12.068259 12040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:12.069116 12035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:12.070317 12035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:12.071558 12035 hybrid_clock.cc:648] HybridClock initialized: now 1790629872071543 us; error 27 us; skew 500 ppm
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:12.074545 12035 init.cc:377] Logged in from keytab as kudu/127.21.140.125@KRBTEST.COM (short username kudu)
I20260928 21:11:12.075544 12035 webserver.cc:460] Webserver started at http://127.21.140.125:42997/ using document root <none> and password file <none>
I20260928 21:11:12.076234 12035 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:12.076305 12035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:12.076488 12035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:12.078666 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data/instance:
uuid: "22717ce9d9c24abebc0bba6818acb73a"
format_stamp: "Formatted at 2026-09-28 21:11:12 on dist-test-slave-d5wc"
server_key: "f28b95e77c1860a08200d2a48d185f5f"
server_key_iv: "53c07e2ab97736692e3198b3ee84a8db"
server_key_version: "encryptionkey@0"
I20260928 21:11:12.079190 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal/instance:
uuid: "22717ce9d9c24abebc0bba6818acb73a"
format_stamp: "Formatted at 2026-09-28 21:11:12 on dist-test-slave-d5wc"
server_key: "f28b95e77c1860a08200d2a48d185f5f"
server_key_iv: "53c07e2ab97736692e3198b3ee84a8db"
server_key_version: "encryptionkey@0"
I20260928 21:11:12.083138 12035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:11:12.085688 12050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:12.087280 12035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:11:12.087433 12035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal
uuid: "22717ce9d9c24abebc0bba6818acb73a"
format_stamp: "Formatted at 2026-09-28 21:11:12 on dist-test-slave-d5wc"
server_key: "f28b95e77c1860a08200d2a48d185f5f"
server_key_iv: "53c07e2ab97736692e3198b3ee84a8db"
server_key_version: "encryptionkey@0"
I20260928 21:11:12.087565 12035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:12.118815 12035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:12.121928 12035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:12.122211 12035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:12.126947 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } attempt: 3
I20260928 21:11:12.127857 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.127538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46756)
0928 21:11:12.127685 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.127760 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 21:11:12.128651 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.128396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46760)
0928 21:11:12.128535 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.128586 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:11:12.128818 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:39473: Network error: Client connection negotiation failed: client connection to 127.21.140.125:39473: connect: Connection refused (error 111)
I20260928 21:11:12.129853 12035 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.125:39473
I20260928 21:11:12.130159 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.125:39473 every 8 connection(s)
I20260928 21:11:12.130874 12035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
I20260928 21:11:12.133683 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:12.135190 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 12035
I20260928 21:11:12.135336 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-1/wal/instance
I20260928 21:11:12.135627 22065 external_mini_cluster.cc:1428] Setting key d8a1bfcd56324a8aa82af88ea7327575
I20260928 21:11:12.138598 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.140.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.124@KRBTEST.COM" created.
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:11:12.155069 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.140637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.21.140.126:44967)
0928 21:11:12.140768 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.140772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.141893 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 21:11:12.142853 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.142857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.142922 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.143032 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.144231 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.144848 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.146057 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.146482 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.146600 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.149819 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.149826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.149832 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.149869 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.152549 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.153190 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.153197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.153200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.153301 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.153723 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.153727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.153729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.154018 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.154147 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.154723 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.154873 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:11:12.155860 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.141118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44967 (local address 127.0.0.1:47316)
0928 21:11:12.141678 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.141720 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.142644 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.143206 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.143213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.143546 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.144079 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.144092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.144994 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.144997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.145854 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.145864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.146675 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.146681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.146916 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.147612 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.147637 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.149633 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.152751 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.152756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.152771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.153073 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.153403 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.153406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.153407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.153616 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.154166 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.154172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.154442 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.154623 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.154915 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":185,"threads_started":1}
I20260928 21:11:12.157560 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab.
Entry for principal kudu/127.21.140.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab.
I20260928 21:11:12.166618 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:34891
--webserver_interface=127.21.140.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab
--principal=kudu/127.21.140.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:44967,127.21.140.125:39473,127.21.140.124:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:11:12.168183 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.158169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46776)
0928 21:11:12.158317 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.158334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.158428 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.159210 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.159213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.159363 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.159459 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.159464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.160717 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.160720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.161355 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.161364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.162139 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.162142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.162262 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.162709 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.162721 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.164143 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.166367 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.166374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.166379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.166632 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.167026 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.167032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.167037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.167142 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.167660 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.167663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.167756 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.167871 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.168009 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:11:12.168295 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.158345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.21.140.125:39473)
0928 21:11:12.158828 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.158839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.158867 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:11:12.158903 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.158907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.159071 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.159185 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.159649 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.160582 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.161489 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.161926 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.162120 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.164270 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.164279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.164291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.164313 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.166260 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.166782 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.166789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.166793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.166898 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.167240 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.167242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.167243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.167494 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.167689 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.167970 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.168122 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":168,"threads_started":1}
I20260928 21:11:12.170358 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:12.171082 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.170832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42010)
0928 21:11:12.170944 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.171023 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:11:12.171300 12052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
I20260928 21:11:12.172159 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.171943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42016)
0928 21:11:12.172063 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.172109 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 21:11:12.172322 12103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:34891: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
I20260928 21:11:12.221583 12103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } attempt: 1
I20260928 21:11:12.222584 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.222224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42028)
0928 21:11:12.222402 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.222477 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 21:11:12.223425 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.223202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42036)
0928 21:11:12.223314 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.223366 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 21:11:12.223604 12103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:34891: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
I20260928 21:11:12.272004 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } attempt: 4
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:11:12.281860 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.272752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.21.140.125:39473)
0928 21:11:12.272906 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.272911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.272930 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:11:12.273046 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.273049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.273101 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.273188 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.273612 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.274913 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.275795 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.276179 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.276296 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.278630 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.278634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.278637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.278673 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.280140 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.280676 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.280678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.280680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.280751 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.281024 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.281027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.281028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.281224 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.281300 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.281634 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.281732 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:11:12.281934 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.272626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46784)
0928 21:11:12.272844 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.272859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.272948 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.273216 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.273218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.273376 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.273490 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.273495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.275073 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.275077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.275644 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.275653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.276333 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.276337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.276439 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.276977 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.276990 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.278471 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.280273 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.280276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.280279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.280577 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.280842 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.280846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.280848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.280940 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.281316 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.281319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.281448 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.281579 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.281756 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 21:11:12.282816 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:12.283530 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.283289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42052)
0928 21:11:12.283422 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.283470 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:11:12.284216 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.284032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42060)
0928 21:11:12.284125 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.284168 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:11:12.284400 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:34891: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
W20260928 21:11:12.285320 12109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:11:12.285637 12109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:12.285703 12109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:12.285774 12109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:12.290133 12109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:12.290216 12109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:12.290261 12109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:12.290297 12109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:12.290334 12109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:12.291785 12103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } attempt: 2
I20260928 21:11:12.292503 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.292256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42064)
0928 21:11:12.292382 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.292437 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:11:12.293215 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.293003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42080)
0928 21:11:12.293109 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.293164 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:11:12.293370 12103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:34891: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
I20260928 21:11:12.296505 12109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:44967,127.21.140.125:39473,127.21.140.124:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab
--principal=kudu/127.21.140.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.140.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:12.297608 12109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:12.298678 12109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:12.307580 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:12.307606 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:12.307631 12109 server_base.cc:1034] running on GCE node
W20260928 21:11:12.307580 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:12.308344 12109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:12.309494 12109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:12.310668 12109 hybrid_clock.cc:648] HybridClock initialized: now 1790629872310655 us; error 25 us; skew 500 ppm
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:12.313627 12109 init.cc:377] Logged in from keytab as kudu/127.21.140.124@KRBTEST.COM (short username kudu)
I20260928 21:11:12.314630 12109 webserver.cc:460] Webserver started at http://127.21.140.124:40829/ using document root <none> and password file <none>
I20260928 21:11:12.315215 12109 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:12.315281 12109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:12.315523 12109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:12.317602 12109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data/instance:
uuid: "36fee7fc4b25406eb18d680668094c31"
format_stamp: "Formatted at 2026-09-28 21:11:12 on dist-test-slave-d5wc"
server_key: "585a4e0bc6c88f0a238d8a86ac5d7636"
server_key_iv: "bc83444ac206d1fcb14272bb1c5384bb"
server_key_version: "encryptionkey@0"
I20260928 21:11:12.318207 12109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal/instance:
uuid: "36fee7fc4b25406eb18d680668094c31"
format_stamp: "Formatted at 2026-09-28 21:11:12 on dist-test-slave-d5wc"
server_key: "585a4e0bc6c88f0a238d8a86ac5d7636"
server_key_iv: "bc83444ac206d1fcb14272bb1c5384bb"
server_key_version: "encryptionkey@0"
I20260928 21:11:12.322229 12109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:11:12.325073 12125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:12.326500 12109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:11:12.326643 12109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal
uuid: "36fee7fc4b25406eb18d680668094c31"
format_stamp: "Formatted at 2026-09-28 21:11:12 on dist-test-slave-d5wc"
server_key: "585a4e0bc6c88f0a238d8a86ac5d7636"
server_key_iv: "bc83444ac206d1fcb14272bb1c5384bb"
server_key_version: "encryptionkey@0"
I20260928 21:11:12.326768 12109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:12.343556 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } attempt: 1
I20260928 21:11:12.344390 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.344104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42092)
0928 21:11:12.344256 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.344313 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 21:11:12.345147 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.344933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42108)
0928 21:11:12.345049 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.345096 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:11:12.345305 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:34891: Network error: Client connection negotiation failed: client connection to 127.21.140.124:34891: connect: Connection refused (error 111)
I20260928 21:11:12.346378 12109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:12.349731 12109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:12.349936 12109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:12.357376 12109 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.124:34891
I20260928 21:11:12.357674 12177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.124:34891 every 8 connection(s)
I20260928 21:11:12.358469 12109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
I20260928 21:11:12.361228 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:12.364200 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 12109
I20260928 21:11:12.364315 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629583469051-22065-0/minicluster-data/master-2/wal/instance
I20260928 21:11:12.364642 22065 external_mini_cluster.cc:1428] Setting key 72706421ece2a52009a7a0ac86775c1c
I20260928 21:11:12.365260 12178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:11:12.379410 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.367355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.21.140.126:44967)
0928 21:11:12.367505 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.367510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.368086 (+ 576us) server_negotiation.cc:407] Connection header received
0928 21:11:12.368165 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.368168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.368217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.368300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.369352 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.370195 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.371676 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.372013 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.372121 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.375241 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.375245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.375248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.375282 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.377234 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.377825 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.377829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.377831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.377924 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.378321 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.378325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.378327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.378554 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.378672 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.379083 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.379220 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:11:12.380304 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.366109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.21.140.124:34891)
0928 21:11:12.366507 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.366525 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.366567 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:11:12.367414 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.367438 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.367801 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.368110 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.369112 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.370419 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.371598 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.372210 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.372389 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.375734 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.375740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.375803 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.375834 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.377460 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.377948 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.377957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.377985 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.378130 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.378431 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.378434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.378436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.378808 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.379026 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.379277 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.379465 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":2432,"thread_start_us":99,"threads_started":1}
I20260928 21:11:12.380304 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.367282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:44967 (local address 127.0.0.1:47326)
0928 21:11:12.367762 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.367809 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.367946 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.368385 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.368391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.368863 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.369193 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.369207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.370320 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.370323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.371555 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.371569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.372159 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.372162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.372373 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.373148 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.373173 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.375041 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.377373 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.377384 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.377394 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.377697 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.378070 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.378074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.378076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.378205 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.378671 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.378680 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.378835 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.379022 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.379401 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":51,"thread_start_us":250,"threads_started":1}
I20260928 21:11:12.381655 12178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:12.384578 12103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } attempt: 3
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:11:12.392266 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.382312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.21.140.125:39473)
0928 21:11:12.382436 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.382442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.382468 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:11:12.382513 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.382516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.382564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.382661 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.383214 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.383757 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.384704 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.385051 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.385147 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.388331 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.388338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.388346 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.388383 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.390222 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.390769 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.390773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.390775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.390866 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.391232 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.391238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.391240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.391518 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.391616 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.391941 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.392071 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:11:12.392325 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.382170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:39473 (local address 127.0.0.1:46786)
0928 21:11:12.382327 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.382340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.382417 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.382704 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.382707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.382886 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.383022 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.383029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.383880 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.383887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.384571 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.384580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.385298 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.385301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.385415 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.386712 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.386726 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.388178 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.390339 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.390342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.390345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.390608 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.391001 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.391006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.391010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.391107 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.391621 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.391626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.391744 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.391869 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.392045 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":20}
I20260928 21:11:12.393043 12178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:12.394234 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.383192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.21.140.126:44967)
0928 21:11:12.383312 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.383316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.383329 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:11:12.383416 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.383419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.383458 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.383561 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.384280 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.384978 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.386234 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.386562 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.386667 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.389573 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.389576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.389578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.389601 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.391901 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.392530 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.392534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.392536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.392626 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.392960 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.392963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.392965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.393226 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.393481 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.393956 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.394122 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:11:12.395227 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.385112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.21.140.124:34891)
0928 21:11:12.385213 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.385218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.385232 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:11:12.385291 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.385293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.385331 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.385404 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.385826 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.386547 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.387776 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.388191 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.388289 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.391099 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.391102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.391104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.391125 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.393399 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.393929 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.393935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.393938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.394056 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.394380 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.394383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.394384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.394591 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.394693 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.394982 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.395083 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":33}
I20260928 21:11:12.395227 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.384990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42128)
0928 21:11:12.385126 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.385142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.385218 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.385422 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.385424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.385611 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.385701 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.385707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.386694 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.386698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.387626 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.387641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.388707 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.388712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.388831 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.389312 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.389333 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.390903 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.393541 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.393544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.393548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.393809 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.394170 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.394175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.394177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.394270 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.394684 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.394689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.394790 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.394928 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.395076 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 21:11:12.398885 12103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: Bootstrap starting.
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629872, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:11:12.402161 12103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:12.403198 12103 log.cc:824] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:12.403275 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.393518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42130)
0928 21:11:12.393633 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.393647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.393716 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.394366 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.394370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.394548 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.394636 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.394645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.396158 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.396162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.396981 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.396999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.397606 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.397610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.397689 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.398249 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.398264 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.399749 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.401576 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.401582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.401585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.401774 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.402093 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.402096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.402098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.402189 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.402755 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.402758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.402838 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.402917 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.403101 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 21:11:12.403862 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.393785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.21.140.124:34891)
0928 21:11:12.394134 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.394145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.394158 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:11:12.394186 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.394189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.394233 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.394739 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.394814 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.396042 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.397120 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.397479 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.397590 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.399884 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.399887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.399890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.399911 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.401465 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.401867 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.401873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.401875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.401956 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.402346 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.402350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.402351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.402639 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.403458 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.403540 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.403682 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260928 21:11:12.405848 12103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: No bootstrap required, opened a new log
I20260928 21:11:12.406143 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.395649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.21.140.125:39473)
0928 21:11:12.395763 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.395767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.395778 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:11:12.395810 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.395813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.395857 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.395937 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.396765 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.397368 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.398451 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.398811 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.398886 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.401805 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.401809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.401812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.401837 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.403698 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.404750 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.404754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.404756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.404849 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.405194 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.405197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.405199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.405425 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.405624 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.405838 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.406007 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:11:12.407310 12178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: Bootstrap starting.
I20260928 21:11:12.409819 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:12.410208 12178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:12.411057 12178 log.cc:824] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:12.410892 12103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.411343 12002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b9373cf3df5b4b7cb66f3109ad893b26"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47330
I20260928 21:11:12.412482 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } attempt: 2
I20260928 21:11:12.411167 12103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22717ce9d9c24abebc0bba6818acb73a, State: Initialized, Role: FOLLOWER
I20260928 21:11:12.413170 12103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.413388 12178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: No bootstrap required, opened a new log
I20260928 21:11:12.414737 12103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:12.417089 12186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.417243 12186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.417701 12178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.417920 12178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fee7fc4b25406eb18d680668094c31, State: Initialized, Role: FOLLOWER
Sep 28 21:11:12 dist-test-slave-d5wc krb5kdc[11947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629871, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:11:12.418309 12178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.420312 12178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:11:12.421346 12197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:12.421434 12197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:12.422935 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.423067 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.420169 12035 master_runner.cc:186] Error getting master registration for 127.21.140.126:44967: OK, instance_id { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" instance_seqno: 1790629871882057 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:11:12.424418 12035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:11:12.426071 12210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:12.426214 12210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:12.425891 12109 master_runner.cc:186] Error getting master registration for 127.21.140.126:44967: OK, instance_id { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" instance_seqno: 1790629871882057 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:11:12.428401 12109 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:11:12.430300 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.412976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.21.140.124:34891)
0928 21:11:12.413079 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.413084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.413176 (+ 92us) server_negotiation.cc:407] Connection header received
0928 21:11:12.413303 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.413307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.413357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.413445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.413981 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.414743 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.415850 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.416202 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.416300 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.419023 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.419028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.419030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.419057 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.420849 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.428960 (+ 8111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.428964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.428966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.429064 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.429367 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.429371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.429373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.429589 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.429715 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.430023 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.430146 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:11:12.430301 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.412917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:34891 (local address 127.0.0.1:42140)
0928 21:11:12.413099 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:12.413115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:12.413214 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:12.413490 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:12.413493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:12.413704 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:12.413832 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.413839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.414878 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.414882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.415705 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:12.415718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.416343 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.416350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:12.416464 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.417081 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:12.417100 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:12.418877 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:12.428533 (+ 9656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.428536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.428538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.428829 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.429156 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:12.429159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:12.429161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:12.429278 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:12.429704 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:12.429707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:12.429819 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 21:11:12.429918 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 21:11:12.430147 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 21:11:12.433569 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26: Bootstrap starting.
I20260928 21:11:12.436381 12028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:12.437255 12028 log.cc:824] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:12.439520 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26: No bootstrap required, opened a new log
I20260928 21:11:12.441941 12028 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.442270 12028 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9373cf3df5b4b7cb66f3109ad893b26, State: Initialized, Role: FOLLOWER
I20260928 21:11:12.442649 12028 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.443195 12213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.443478 12213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.444060 12028 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:12.444097 12002 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:11:12.444185 12002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:12.445582 12002 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.446640 12002 leader_election.cc:288] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [CANDIDATE]: Term 1 election: Requested vote from peers 22717ce9d9c24abebc0bba6818acb73a (127.21.140.125:39473), 36fee7fc4b25406eb18d680668094c31 (127.21.140.124:34891)
I20260928 21:11:12.449856 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "22717ce9d9c24abebc0bba6818acb73a"
I20260928 21:11:12.450157 12077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:12.451637 12077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9373cf3df5b4b7cb66f3109ad893b26 in term 1.
W20260928 21:11:12.452531 12224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:12.452600 12224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:12.452658 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36fee7fc4b25406eb18d680668094c31"
I20260928 21:11:12.452901 12152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:12.454493 12152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9373cf3df5b4b7cb66f3109ad893b26 in term 1.
I20260928 21:11:12.455235 11977 leader_election.cc:302] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22717ce9d9c24abebc0bba6818acb73a, b9373cf3df5b4b7cb66f3109ad893b26; no voters:
I20260928 21:11:12.459920 11960 master_runner.cc:263] No leader master found from master b9373cf3df5b4b7cb66f3109ad893b26
I20260928 21:11:12.460001 11960 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:11:12.460201 12213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:12.460520 12213 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [term 1 LEADER]: Becoming Leader. State: Replica: b9373cf3df5b4b7cb66f3109ad893b26, State: Running, Role: LEADER
I20260928 21:11:12.460984 12213 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } }
I20260928 21:11:12.463028 12213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9373cf3df5b4b7cb66f3109ad893b26. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.463215 12213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:12.463817 12235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:12.464838 12235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:12.469163 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.449253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.21.140.126:44967)
0928 21:11:12.449360 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.449365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.452924 (+ 3559us) server_negotiation.cc:407] Connection header received
0928 21:11:12.453152 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.453155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.453211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.453271 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.454408 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.455140 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.456255 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.456565 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.456642 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.458628 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.458632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.458635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.458662 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.460442 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.460990 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.460996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.461001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.461087 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.461398 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.461402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.461403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.461654 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.461770 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.468686 (+ 6916us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.468850 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:11:12.473387 12152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [term 1 FOLLOWER]: Refusing update from remote peer b9373cf3df5b4b7cb66f3109ad893b26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:11:12.474221 12233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:11:12.477471 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.454970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.21.140.124:34891)
0928 21:11:12.455075 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.455079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.468520 (+ 13441us) server_negotiation.cc:407] Connection header received
0928 21:11:12.468690 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.468693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.468745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.468799 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.469667 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.470448 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.471568 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.471900 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.471983 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.473698 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.473702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.473705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.473731 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.475390 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.475872 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.475876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.475878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.475972 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.476255 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.476258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.476260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.476464 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.476597 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.477175 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.477294 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:11:12.479761 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:12.454320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.21.140.125:39473)
0928 21:11:12.454434 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.454438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.464932 (+ 10494us) server_negotiation.cc:407] Connection header received
0928 21:11:12.465125 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.465128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.465180 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.465231 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.466174 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.466969 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.468359 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:12.468732 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:12.468810 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:12.470735 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:12.470739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:12.470742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:12.470770 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:12.472571 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.478356 (+ 5785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.478360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.478362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.478463 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:12.478763 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:12.478766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:12.478768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:12.479021 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:12.479148 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:12.479481 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 21:11:12.479605 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:11:12.480340 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9373cf3df5b4b7cb66f3109ad893b26. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.480511 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.483052 12077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [term 1 FOLLOWER]: Refusing update from remote peer b9373cf3df5b4b7cb66f3109ad893b26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:11:12.483736 12233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:11:12.489441 12186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9373cf3df5b4b7cb66f3109ad893b26. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.489578 12186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.496001 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.496150 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:12.497437 12186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.497558 12186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.499713 12236 mvcc.cc:205] Tried to move back new op lower bound from 7334419957642092544 to 7334419957604708352. Current Snapshot: MvccSnapshot[applied={T|T < 7334419957642092544}]
I20260928 21:11:12.502614 12239 mvcc.cc:205] Tried to move back new op lower bound from 7334419957642092544 to 7334419957604708352. Current Snapshot: MvccSnapshot[applied={T|T < 7334419957642092544}]
I20260928 21:11:12.503463 12213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.503582 12213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:12.504777 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9373cf3df5b4b7cb66f3109ad893b26" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 44967 } } peers { permanent_uuid: "22717ce9d9c24abebc0bba6818acb73a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 39473 } } peers { permanent_uuid: "36fee7fc4b25406eb18d680668094c31" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 34891 } } }
I20260928 21:11:12.504912 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:12.506750 12237 mvcc.cc:205] Tried to move back new op lower bound from 7334419957642092544 to 7334419957604708352. Current Snapshot: MvccSnapshot[applied={T|T < 7334419957642092544}]
I20260928 21:11:12.510763 12235 catalog_manager.cc:1347] Generated new cluster ID: 0db7a2acb3d6475ca0b3466f6f21ce02
I20260928 21:11:12.510828 12235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:12.544763 12235 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:12.546502 12235 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:12.593437 12235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9373cf3df5b4b7cb66f3109ad893b26: Generated new TSK 0
I20260928 21:11:12.594394 12235 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:11:12.635208 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.631726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.21.140.125:39473)
0928 21:11:12.631862 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.631867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.633877 (+ 2010us) server_negotiation.cc:407] Connection header received
0928 21:11:12.634185 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.634189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.634248 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.634310 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.635057 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46802: BlockingRecv error: recv got EOF from 127.0.0.1:46802 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 21:11:12.636958 12183 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.632559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.21.140.124:34891)
0928 21:11:12.632706 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.632711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.635934 (+ 3223us) server_negotiation.cc:407] Connection header received
0928 21:11:12.636138 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.636141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.636201 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.636259 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.636809 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42152: BlockingRecv error: recv got EOF from 127.0.0.1:42152 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:11:12.638605 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.633225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.21.140.126:44967)
0928 21:11:12.633342 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.633346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.637612 (+ 4266us) server_negotiation.cc:407] Connection header received
0928 21:11:12.637801 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.637805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.637858 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.637930 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.638479 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47354: BlockingRecv error: recv got EOF from 127.0.0.1:47354 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 21:11:12.643920 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.642019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.21.140.125:39473)
0928 21:11:12.642146 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.642151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.642784 (+ 633us) server_negotiation.cc:407] Connection header received
0928 21:11:12.643046 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.643049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.643135 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.643201 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.643788 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46804: BlockingRecv error: recv got EOF from 127.0.0.1:46804 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 21:11:12.648974 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.647973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.21.140.125:39473)
0928 21:11:12.648083 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.648087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.648101 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:11:12.648153 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.648157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.648206 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.648310 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.648857 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46808: BlockingRecv error: recv got EOF from 127.0.0.1:46808 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:11:12.654455 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.653203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.21.140.125:39473)
0928 21:11:12.653309 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.653313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.653541 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:11:12.653680 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.653683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.653738 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.653820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.654332 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46822: BlockingRecv error: recv got EOF from 127.0.0.1:46822 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:11:12.662058 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.660854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.21.140.125:39473)
0928 21:11:12.661047 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.661052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.661127 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:11:12.661260 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.661262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.661309 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.661383 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.661902 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46824: BlockingRecv error: recv got EOF from 127.0.0.1:46824 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 21:11:12.673269 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.672089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.21.140.125:39473)
0928 21:11:12.672220 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.672224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.672344 (+ 120us) server_negotiation.cc:407] Connection header received
0928 21:11:12.672523 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.672526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.672581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.672675 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.673155 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46836: BlockingRecv error: recv got EOF from 127.0.0.1:46836 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 21:11:12.692786 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.691556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.21.140.125:39473)
0928 21:11:12.691730 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.691735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.691847 (+ 112us) server_negotiation.cc:407] Connection header received
0928 21:11:12.692026 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.692030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.692084 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.692178 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.692655 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46844: BlockingRecv error: recv got EOF from 127.0.0.1:46844 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 21:11:12.728284 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.727086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.21.140.125:39473)
0928 21:11:12.727249 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.727253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.727389 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:11:12.727464 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.727467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.727520 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.727609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.728172 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46860: BlockingRecv error: recv got EOF from 127.0.0.1:46860 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 21:11:12.795753 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.794609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.21.140.125:39473)
0928 21:11:12.794773 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.794777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.794880 (+ 103us) server_negotiation.cc:407] Connection header received
0928 21:11:12.795004 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.795006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.795059 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.795159 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.795657 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46876: BlockingRecv error: recv got EOF from 127.0.0.1:46876 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 21:11:12.927198 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:12.925913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.21.140.125:39473)
0928 21:11:12.926144 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:12.926149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:12.926257 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:11:12.926382 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:12.926385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:12.926433 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:12.926522 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:12.927086 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46888: BlockingRecv error: recv got EOF from 127.0.0.1:46888 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260928 21:11:13.187574 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:13.186366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.21.140.125:39473)
0928 21:11:13.186533 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:13.186537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:13.186555 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:11:13.186695 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:13.186698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:13.186759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:13.186855 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:13.187438 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46896: BlockingRecv error: recv got EOF from 127.0.0.1:46896 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 21:11:13.423306 12197 catalog_manager.cc:1259] Loaded cluster ID: 0db7a2acb3d6475ca0b3466f6f21ce02
I20260928 21:11:13.423429 12197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: loading cluster ID for follower catalog manager: success
I20260928 21:11:13.425551 12197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: acquiring CA information for follower catalog manager: success
I20260928 21:11:13.427265 12197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22717ce9d9c24abebc0bba6818acb73a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:11:13.427835 12210 catalog_manager.cc:1259] Loaded cluster ID: 0db7a2acb3d6475ca0b3466f6f21ce02
I20260928 21:11:13.427920 12210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: loading cluster ID for follower catalog manager: success
I20260928 21:11:13.429915 12210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: acquiring CA information for follower catalog manager: success
I20260928 21:11:13.431766 12210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36fee7fc4b25406eb18d680668094c31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:11:13.707648 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:13.702445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.21.140.125:39473)
0928 21:11:13.702851 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:13.702858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:13.702892 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:11:13.702984 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:13.702988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:13.703284 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:13.703464 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:13.703872 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.704461 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.705565 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.705996 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.706131 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:13.706505 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:13.707135 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:13.707250 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:13.707314 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:11:13.707393 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260928 21:11:13.716475 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:13.711686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.21.140.124:34891)
0928 21:11:13.712004 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:13.712008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:13.712024 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:11:13.712164 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:13.712167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:13.712334 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:13.712463 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:13.712709 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.713325 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.714410 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.714669 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.714764 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:13.715229 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:13.715892 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:13.716099 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:13.716137 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:11:13.716227 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":145,"threads_started":1}
I20260928 21:11:13.725562 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:13.720746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.21.140.125:39473)
0928 21:11:13.720929 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:13.720934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:13.721534 (+ 600us) server_negotiation.cc:407] Connection header received
0928 21:11:13.721811 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:13.721815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:13.721890 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:13.721949 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:13.722521 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.723372 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.724295 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.724596 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.724710 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:13.724968 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:13.725093 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:13.725174 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:13.725294 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:11:13.725370 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:11:13.726598 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:13.720765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.21.140.126:44967)
0928 21:11:13.721129 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:13.721137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:13.721242 (+ 105us) server_negotiation.cc:407] Connection header received
0928 21:11:13.721536 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:13.721540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:13.721817 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:13.722003 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:13.722282 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.723058 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.724574 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.724973 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.725090 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:13.725457 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:13.726021 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:13.726095 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:13.726308 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 21:11:13.726402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":99,"threads_started":1}
I20260928 21:11:13.726619 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:13.721305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.21.140.124:34891)
0928 21:11:13.721501 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:13.721506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:13.721741 (+ 235us) server_negotiation.cc:407] Connection header received
0928 21:11:13.721833 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:13.721839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:13.721909 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:13.722038 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:13.722404 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.723260 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.725229 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:13.725561 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:13.725681 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:13.726041 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:13.726149 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:13.726201 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:13.726381 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 21:11:13.726473 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:11:13.731278 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 11960
I20260928 21:11:13.739305 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 12035
I20260928 21:11:13.747406 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 12109
2026-09-28T21:11:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3229 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:13 dist-test-slave-d5wc krb5kdc[12326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:13 dist-test-slave-d5wc krb5kdc[12326](info): set up 2 sockets
Sep 28 21:11:13 dist-test-slave-d5wc krb5kdc[12326](info): commencing operation
krb5kdc: starting...
W20260928 21:11:14.808096 22065 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/test-user.keytab.
Sep 28 21:11:14 dist-test-slave-d5wc krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.140.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
Entry for principal kudu/127.21.140.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab.
I20260928 21:11:14.936260 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:36041
--webserver_interface=127.21.140.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:36041,127.21.140.125:42595,127.21.140.124:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:15.051259 12339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:11:15.051676 12339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:15.051748 12339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:15.051818 12339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:15.056038 12339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:15.056108 12339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:15.056129 12339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:15.056149 12339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:15.056164 12339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:15.061831 12339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:36041,127.21.140.125:42595,127.21.140.124:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.126.keytab
--principal=kudu/127.21.140.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.126:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.140.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:15.063056 12339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:15.064132 12339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:15.073117 12347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:15.073114 12345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:15.073133 12344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:15.073447 12339 server_base.cc:1034] running on GCE node
I20260928 21:11:15.074234 12339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:15.075624 12339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:15.076845 12339 hybrid_clock.cc:648] HybridClock initialized: now 1790629875076830 us; error 27 us; skew 500 ppm
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:15.080072 12339 init.cc:377] Logged in from keytab as kudu/127.21.140.126@KRBTEST.COM (short username kudu)
I20260928 21:11:15.081140 12339 webserver.cc:460] Webserver started at http://127.21.140.126:36217/ using document root <none> and password file <none>
I20260928 21:11:15.081773 12339 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:15.081875 12339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:15.082150 12339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:15.084309 12339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data/instance:
uuid: "2ced7e3ccdfa4f9984f09f74ed94821a"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "c911981618520bde794896bf5455b647"
server_key_iv: "b0a8312d2afb6d09c21fe8438ab30ccf"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.084933 12339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal/instance:
uuid: "2ced7e3ccdfa4f9984f09f74ed94821a"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "c911981618520bde794896bf5455b647"
server_key_iv: "b0a8312d2afb6d09c21fe8438ab30ccf"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.089007 12339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 21:11:15.091867 12354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:15.093484 12339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:11:15.093657 12339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal
uuid: "2ced7e3ccdfa4f9984f09f74ed94821a"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "c911981618520bde794896bf5455b647"
server_key_iv: "b0a8312d2afb6d09c21fe8438ab30ccf"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.093791 12339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:15.107877 12339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:15.110929 12339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:15.111150 12339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:15.118408 12339 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.126:36041
I20260928 21:11:15.118698 12406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.126:36041 every 8 connection(s)
I20260928 21:11:15.119446 12339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:15.122306 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:15.123008 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 12339
I20260928 21:11:15.123149 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-0/wal/instance
I20260928 21:11:15.123486 22065 external_mini_cluster.cc:1428] Setting key e33bb23c327821f45362bc957e7f9c6d
I20260928 21:11:15.127133 12407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.140.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.125@KRBTEST.COM" created.
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:11:15.144299 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.129764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:36041 (local address 127.0.0.1:58964)
0928 21:11:15.130396 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.130433 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.131307 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.132082 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.132088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.132555 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.132822 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.132836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.134125 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.134128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.135264 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.135279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.136117 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.136120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.136285 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.137051 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.137075 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.138955 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.141157 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.141168 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.141178 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.141407 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.141746 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.141750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.141752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.141863 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.142542 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.142548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.142693 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.142859 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.143246 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":164,"spinlock_wait_cycles":14592,"thread_start_us":175,"threads_started":1}
I20260928 21:11:15.144299 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.129764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.21.140.126:36041)
0928 21:11:15.130249 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.130269 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.130552 (+ 283us) server_negotiation.cc:407] Connection header received
0928 21:11:15.131638 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.131643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.131891 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.132089 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.132979 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.133912 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.135441 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.135928 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.136171 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.139137 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.139143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.139153 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.139175 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.141012 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.141524 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.141530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.141532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.141641 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.142015 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.142020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.142022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.142417 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.142573 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.142932 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.143195 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":331,"thread_start_us":168,"threads_started":1}
I20260928 21:11:15.147228 12407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:15.148124 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.147784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36560)
0928 21:11:15.147966 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.148050 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260928 21:11:15.148429 12356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
I20260928 21:11:15.149488 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.149168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36566)
0928 21:11:15.149333 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.149382 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 21:11:15.149665 12407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:42595: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab.
Entry for principal kudu/127.21.140.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab.
I20260928 21:11:15.154078 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:42595
--webserver_interface=127.21.140.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab
--principal=kudu/127.21.140.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:36041,127.21.140.125:42595,127.21.140.124:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:11:15.198905 12407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } attempt: 1
I20260928 21:11:15.199811 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.199501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36568)
0928 21:11:15.199663 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.199735 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 21:11:15.200536 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.200295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36578)
0928 21:11:15.200419 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.200473 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:11:15.200664 12407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:42595: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
W20260928 21:11:15.265734 12414 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:11:15.266048 12414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:15.266073 12414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:15.266178 12414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:11:15.268852 12407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } attempt: 2
I20260928 21:11:15.269748 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.269409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36584)
0928 21:11:15.269610 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.269675 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 21:11:15.270524 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.270289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36586)
0928 21:11:15.270401 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.270453 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:11:15.270659 12414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:15.270656 12407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.125:42595: Network error: Client connection negotiation failed: client connection to 127.21.140.125:42595: connect: Connection refused (error 111)
W20260928 21:11:15.270731 12414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:15.270776 12414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:15.270831 12414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:15.270869 12414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:15.276468 12414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:36041,127.21.140.125:42595,127.21.140.124:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.125.keytab
--principal=kudu/127.21.140.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.125:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.140.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:15.277634 12414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:15.278730 12414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:15.287319 12420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:15.287367 12422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:15.287319 12419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:15.287341 12414 server_base.cc:1034] running on GCE node
I20260928 21:11:15.288264 12414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:15.289414 12414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:15.290601 12414 hybrid_clock.cc:648] HybridClock initialized: now 1790629875290555 us; error 60 us; skew 500 ppm
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:15.293537 12414 init.cc:377] Logged in from keytab as kudu/127.21.140.125@KRBTEST.COM (short username kudu)
I20260928 21:11:15.294467 12414 webserver.cc:460] Webserver started at http://127.21.140.125:42233/ using document root <none> and password file <none>
I20260928 21:11:15.294986 12414 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:15.295048 12414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:15.295213 12414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:15.297334 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data/instance:
uuid: "34a0b04c5f144e9893c6466dcd1adc3a"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "f542ecf9ef31ba4c483133982927df22"
server_key_iv: "68479792dc7856ed56c668578926d310"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.297922 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal/instance:
uuid: "34a0b04c5f144e9893c6466dcd1adc3a"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "f542ecf9ef31ba4c483133982927df22"
server_key_iv: "68479792dc7856ed56c668578926d310"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.301911 12414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 21:11:15.304661 12429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:15.306043 12414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:11:15.306186 12414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal
uuid: "34a0b04c5f144e9893c6466dcd1adc3a"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "f542ecf9ef31ba4c483133982927df22"
server_key_iv: "68479792dc7856ed56c668578926d310"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.306319 12414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:15.336108 12414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:15.339340 12414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:15.339581 12414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:15.346762 12414 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.125:42595
I20260928 21:11:15.347112 12481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.125:42595 every 8 connection(s)
I20260928 21:11:15.347775 12414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/data/info.pb
I20260928 21:11:15.350718 12482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:15.351068 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 12414
I20260928 21:11:15.351217 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-1/wal/instance
I20260928 21:11:15.351539 22065 external_mini_cluster.cc:1428] Setting key df68c6d3c51b9066621b19b2030df508
I20260928 21:11:15.355234 12482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.140.124@KRBTEST.COM; defaulting to no policy
I20260928 21:11:15.361845 12407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.140.124@KRBTEST.COM" created.
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:11:15.372225 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.357309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.21.140.126:36041)
0928 21:11:15.357499 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.357502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.358582 (+ 1080us) server_negotiation.cc:407] Connection header received
0928 21:11:15.359604 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.359608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.359674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.359789 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.361019 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.361558 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.363232 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.363603 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.363727 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.366907 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.366911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.366919 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.366943 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.369838 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.370563 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.370566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.370570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.370650 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.370999 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.371002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.371011 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.371193 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.371300 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.371884 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.372018 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:11:15.373099 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.357823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:36041 (local address 127.0.0.1:58976)
0928 21:11:15.358385 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.358431 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.359312 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.359977 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.359986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.360333 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.360805 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.360819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.361694 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.361697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.363106 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.363118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.363765 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.363772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.364071 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.364797 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.364825 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.366727 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.369955 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.369995 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.370006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.370425 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.370743 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.370747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.370749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.370893 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.371300 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.371308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.371625 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.371812 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.372136 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":3,"thread_start_us":202,"threads_started":1}
I20260928 21:11:15.373234 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.362381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36600)
0928 21:11:15.362580 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.362593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.362665 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.363354 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.363357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.363546 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.363661 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.363670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.364908 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.364914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.365990 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.366003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.366761 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.366765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.366874 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.367384 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.367396 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.368859 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.371407 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.371411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.371413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.371692 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.372073 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.372079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.372084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.372199 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.372789 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.372795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.372903 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.372987 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.373089 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 21:11:15.373831 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.362515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.21.140.125:42595)
0928 21:11:15.362888 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.362902 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.362926 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:11:15.362971 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.362976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.363187 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.363363 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.363785 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.364760 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.366196 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.366625 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.366747 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.369026 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.369036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.369054 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.369088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.371263 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.371846 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.371853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.371855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.371946 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.372334 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.372337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.372339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.372655 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.373425 (+ 770us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.373539 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.373690 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":249,"thread_start_us":173,"threads_started":1}
I20260928 21:11:15.374585 12482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:15.375726 12407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:15.376377 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.376152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54072)
0928 21:11:15.376262 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.376314 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:11:15.377094 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.376826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54084)
0928 21:11:15.376993 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.377042 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 21:11:15.377204 12407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:36253: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.140.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab.
Entry for principal kudu/127.21.140.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab.
I20260928 21:11:15.381889 22065 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczrM8B/build/debug/bin/kudu
/tmp/dist-test-taskczrM8B/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:36253
--webserver_interface=127.21.140.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab
--principal=kudu/127.21.140.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.140.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.140.126:36041,127.21.140.125:42595,127.21.140.124:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:11:15.386643 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.375142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.21.140.125:42595)
0928 21:11:15.375246 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.375250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.375375 (+ 125us) server_negotiation.cc:407] Connection header received
0928 21:11:15.375493 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.375496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.375541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.375624 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.378700 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.379348 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.380263 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.380701 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.380809 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.383274 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.383281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.383283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.383311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.384858 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.385338 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.385340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.385342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.385414 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.385717 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.385720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.385721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.385891 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.386025 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.386321 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.386503 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43}
I20260928 21:11:15.386873 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.375106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36610)
0928 21:11:15.375287 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.375304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.375399 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.378189 (+ 2790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.378193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.378354 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.378552 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.378560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.379490 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.379493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.380110 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.380120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.380834 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.380840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.380961 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.381473 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.381487 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.383115 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.384947 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.384953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.384958 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.385229 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.385509 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.385515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.385520 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.385621 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.386021 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.386026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.386145 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.386274 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.386599 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":136}
I20260928 21:11:15.387609 12482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:15.388321 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.388082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54100)
0928 21:11:15.388195 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.388270 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 21:11:15.389111 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
I20260928 21:11:15.390093 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.389846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54116)
0928 21:11:15.389951 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.390047 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 21:11:15.390331 12482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:36253: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
I20260928 21:11:15.408393 12407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } attempt: 1
I20260928 21:11:15.409343 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.409017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54118)
0928 21:11:15.409179 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.409242 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 21:11:15.410252 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.410036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54128)
0928 21:11:15.410144 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.410195 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:11:15.410377 12407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:36253: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
I20260928 21:11:15.439623 12482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } attempt: 1
I20260928 21:11:15.440582 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.440215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54136)
0928 21:11:15.440391 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.440469 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 21:11:15.441346 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.441138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54138)
0928 21:11:15.441246 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.441298 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:11:15.441478 12482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:36253: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
I20260928 21:11:15.485569 12407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } attempt: 2
I20260928 21:11:15.486509 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.486160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54142)
0928 21:11:15.486330 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.486393 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 21:11:15.487275 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.487060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54156)
0928 21:11:15.487170 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.487223 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 21:11:15.487408 12407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:36253: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
W20260928 21:11:15.497424 12488 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:11:15.497758 12488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:15.497797 12488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:15.497838 12488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:15.501937 12488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:15.502060 12488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:15.502084 12488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:15.502101 12488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:15.502116 12488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:15.507505 12488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.140.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.140.126:36041,127.21.140.125:42595,127.21.140.124:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/krb5kdc/kudu_127.21.140.124.keytab
--principal=kudu/127.21.140.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.140.124:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.140.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:15.508515 12488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:15.509510 12488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:11:15.509661 12482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } attempt: 2
I20260928 21:11:15.510561 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.510200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54166)
0928 21:11:15.510367 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.510481 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 21:11:15.511310 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.511093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54172)
0928 21:11:15.511212 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.511261 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 21:11:15.511520 12482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.140.124:36253: Network error: Client connection negotiation failed: client connection to 127.21.140.124:36253: connect: Connection refused (error 111)
I20260928 21:11:15.518440 12488 server_base.cc:1034] running on GCE node
W20260928 21:11:15.518409 12495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:15.518404 12494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:15.518404 12497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:15.519212 12488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:15.520313 12488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:15.521541 12488 hybrid_clock.cc:648] HybridClock initialized: now 1790629875521528 us; error 39 us; skew 500 ppm
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:15.530915 12488 init.cc:377] Logged in from keytab as kudu/127.21.140.124@KRBTEST.COM (short username kudu)
I20260928 21:11:15.531889 12488 webserver.cc:460] Webserver started at http://127.21.140.124:37731/ using document root <none> and password file <none>
I20260928 21:11:15.532477 12488 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:15.532573 12488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:15.532815 12488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:15.534961 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data/instance:
uuid: "836c8d46b4524e61b21573ead0b4ce69"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "e5479f273d53a2b3d7072ea538ed2f16"
server_key_iv: "a875a8154080064ce951061f5f0625d0"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.535554 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal/instance:
uuid: "836c8d46b4524e61b21573ead0b4ce69"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "e5479f273d53a2b3d7072ea538ed2f16"
server_key_iv: "a875a8154080064ce951061f5f0625d0"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.539397 12488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:11:15.542141 12504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:15.543457 12488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:11:15.543625 12488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data,/tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal
uuid: "836c8d46b4524e61b21573ead0b4ce69"
format_stamp: "Formatted at 2026-09-28 21:11:15 on dist-test-slave-d5wc"
server_key: "e5479f273d53a2b3d7072ea538ed2f16"
server_key_iv: "a875a8154080064ce951061f5f0625d0"
server_key_version: "encryptionkey@0"
I20260928 21:11:15.543752 12488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:15.569351 12488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:15.572826 12488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:15.573060 12488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:15.580178 12488 rpc_server.cc:307] RPC server started. Bound to: 127.21.140.124:36253
I20260928 21:11:15.580514 12556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.140.124:36253 every 8 connection(s)
I20260928 21:11:15.581195 12488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/data/info.pb
I20260928 21:11:15.584077 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:15.586571 12407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } attempt: 3
I20260928 21:11:15.588102 12557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:15.590090 22065 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczrM8B/build/debug/bin/kudu as pid 12488
I20260928 21:11:15.590240 22065 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczrM8B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629583469051-22065-0/minicluster-data/master-2/wal/instance
I20260928 21:11:15.590606 22065 external_mini_cluster.cc:1428] Setting key cf6db50d17798899fd2d048f12c7053c
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.126@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:11:15.601779 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.590620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.21.140.126:36041)
0928 21:11:15.590741 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.590745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.591120 (+ 375us) server_negotiation.cc:407] Connection header received
0928 21:11:15.591304 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.591307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.591350 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.591398 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.592367 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.592891 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.593984 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.594320 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.594424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.597626 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.597633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.597638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.597668 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.599452 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.600046 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.600053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.600058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.600155 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.600629 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.600634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.600636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.600861 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.600990 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.601406 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.601564 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":50}
I20260928 21:11:15.602116 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.587080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54178)
0928 21:11:15.587214 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.587232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.587328 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.589944 (+ 2616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.589948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.590185 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.590355 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.590364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.592270 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.592275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.593158 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.593168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.594239 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.594245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.594396 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.595004 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.595022 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.596589 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.599733 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.599736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.599738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.599985 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.600338 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.600341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.600343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.600440 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.601275 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.601279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.601380 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.601472 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.601642 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":74,"spinlock_wait_cycles":5376}
I20260928 21:11:15.602754 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.590088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.126:36041 (local address 127.0.0.1:58978)
0928 21:11:15.590991 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.591040 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.591201 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.591567 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.591571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.592014 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.592235 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.592246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.593032 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.593036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.593807 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.593817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.594472 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.594476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.594627 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.595236 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.595256 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.597480 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.599576 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.599584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.599592 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.599931 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.600254 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.600257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.600258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.600441 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.600971 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.600978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.601138 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.601323 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.601661 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":56,"thread_start_us":153,"threads_started":1}
I20260928 21:11:15.602694 12482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } attempt: 3
I20260928 21:11:15.602774 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.587819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.21.140.124:36253)
0928 21:11:15.588250 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.588270 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.588314 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:11:15.589170 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.589192 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.589613 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.590031 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.590512 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.592076 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.593278 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.594091 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.594238 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.596738 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.596744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.596757 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.596788 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.599616 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.600088 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.600096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.600100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.600235 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.600555 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.600561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.600564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.601148 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.601560 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.601681 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.601826 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260928 21:11:15.604256 12557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:11:15.605950 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.591224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.21.140.124:36253)
0928 21:11:15.591498 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.591508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.591776 (+ 268us) server_negotiation.cc:407] Connection header received
0928 21:11:15.591931 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.591934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.591981 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.592078 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.592981 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.593784 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.595222 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.595575 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.595686 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.598664 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.598669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.598674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.598703 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.602685 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.603241 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.603245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.603246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.603329 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.603623 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.603627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.603628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.603832 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.604013 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.605682 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.605796 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260928 21:11:15.607836 12407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a: Bootstrap starting.
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:11:15.611095 12407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:15.612182 12407 log.cc:824] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.126@KRBTEST.COM
I20260928 21:11:15.614598 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.604721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.125:42595 (local address 127.0.0.1:36614)
0928 21:11:15.605092 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.605107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.605209 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.605549 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.605552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.605738 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.605859 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.605866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.607092 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.607095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.607887 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.607899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.608576 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.608579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.608676 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.609200 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.609215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.610799 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.612843 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.612847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.612849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.613088 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.613467 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.613473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.613478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.613583 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.614082 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.614085 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.614178 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.614294 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.614431 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260928 21:11:15.614689 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.604840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.21.140.125:42595)
0928 21:11:15.604946 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.604952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.605155 (+ 203us) server_negotiation.cc:407] Connection header received
0928 21:11:15.605308 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.605311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.605354 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.605400 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.606032 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.606746 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.608029 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.608339 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.608414 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.610919 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.610925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.610931 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.610965 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.612661 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.613198 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.613201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.613203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.613287 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.613704 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.613707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.613709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.613913 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.614035 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.614373 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.614510 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:11:15.615017 12407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a: No bootstrap required, opened a new log
I20260928 21:11:15.615911 12557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.125@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
I20260928 21:11:15.618431 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.607844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.21.140.126:36041)
0928 21:11:15.607959 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.607963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.608033 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:11:15.608164 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.608167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.608211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.608301 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.608973 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.609554 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.610785 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.611076 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.611131 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.613911 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.613915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.613918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.613944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.616116 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.616903 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.616907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.616909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.616994 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.617403 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.617406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.617408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.617635 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.617773 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.618163 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.618278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:11:15.619793 12407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.620330 12407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ced7e3ccdfa4f9984f09f74ed94821a, State: Initialized, Role: FOLLOWER
I20260928 21:11:15.620759 12407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.621450 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.603131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54186)
0928 21:11:15.603255 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.603271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.603356 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.604485 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.604489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.604786 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.604911 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.604918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.613411 (+ 8493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.613415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.614546 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.614556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.615427 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.615432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.615523 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.616000 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.616019 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.617406 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.619501 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.619504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.619506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.619741 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.620057 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.620060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.620062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.620146 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.620647 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.620650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.620750 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.621132 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.621287 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:11:15.622118 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.603866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.21.140.124:36253)
0928 21:11:15.604154 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.604163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.604176 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:11:15.604208 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.604210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.604253 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.604449 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.605024 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.613281 (+ 8257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.614681 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.615055 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.616611 (+ 1556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.617549 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.617553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.617556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.617580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.619376 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.619857 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.619860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.619862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.619956 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.620247 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.620249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.620251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.620539 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.621637 (+ 1098us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.621715 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.621841 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
I20260928 21:11:15.622435 12407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629875, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.140.124@KRBTEST.COM for kudu/127.21.140.124@KRBTEST.COM
Sep 28 21:11:15 dist-test-slave-d5wc krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.140.125@KRBTEST.COM
I20260928 21:11:15.626308 12482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: Bootstrap starting.
I20260928 21:11:15.627024 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.616405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.140.124:36253 (local address 127.0.0.1:54188)
0928 21:11:15.616526 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:15.616555 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:15.616629 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:15.617590 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:15.617593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:15.617813 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:15.617927 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.617935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.619056 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.619059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.619875 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:15.619883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.620601 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.620608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:15.620681 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.621129 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:15.621144 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:15.623208 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:15.625463 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.625467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.625469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.625738 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.626076 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:15.626079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:15.626080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:15.626185 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:15.626587 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:15.626589 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:15.626664 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 21:11:15.626756 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:11:15.626881 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:11:15.627506 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.617213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.21.140.124:36253)
0928 21:11:15.617322 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.617326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.617339 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:11:15.617374 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.617376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.617414 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.617497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.618079 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.618939 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.619987 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.620366 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.620507 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.623345 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.623348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.623351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.623372 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.625334 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.625854 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.625857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.625858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.625937 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.626281 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.626284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.626286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.626469 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.626555 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.627249 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.627376 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":43}
W20260928 21:11:15.629057 12576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:15.629146 12576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:15.631023 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: Bootstrap starting.
I20260928 21:11:15.631835 12482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:15.632781 12482 log.cc:824] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:15.624938 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.633265 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:15.633750 12557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:15.634411 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.620153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.21.140.125:42595)
0928 21:11:15.620261 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.620265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.620543 (+ 278us) server_negotiation.cc:407] Connection header received
0928 21:11:15.620710 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.620713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.620758 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.620809 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.621670 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.622372 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.623489 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.623827 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.623905 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.626986 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.626990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.626993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.627018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.629140 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.629583 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.629586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.629588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.629671 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.629984 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.629990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.629995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.630238 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.630419 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.634132 (+ 3713us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.634259 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:11:15.634721 12557 log.cc:824] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:15.633388 12339 master_runner.cc:186] Error getting master registration for 127.21.140.125:42595: OK, instance_id { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" instance_seqno: 1790629875344690 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:11:15.636607 12339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:11:15.636911 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: No bootstrap required, opened a new log
I20260928 21:11:15.639433 22065 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:11:15.640933 12381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58984
I20260928 21:11:15.641198 12381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:11:15.641340 12381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:15.641881 12557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.642196 12557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836c8d46b4524e61b21573ead0b4ce69, State: Initialized, Role: FOLLOWER
I20260928 21:11:15.642560 12557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.642710 12381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.643671 12381 leader_election.cc:288] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [CANDIDATE]: Term 1 election: Requested vote from peers 34a0b04c5f144e9893c6466dcd1adc3a (127.21.140.125:42595), 836c8d46b4524e61b21573ead0b4ce69 (127.21.140.124:36253)
I20260928 21:11:15.643899 12557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:15.644044 12482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: No bootstrap required, opened a new log
I20260928 21:11:15.647809 12482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.648025 12482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a0b04c5f144e9893c6466dcd1adc3a, State: Initialized, Role: FOLLOWER
I20260928 21:11:15.648175 12580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.648291 12580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:15.648337 12482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.649641 12482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:11:15.649859 12591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 21:11:15.650151 12593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.650266 12593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:11:15.649945 12591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:15.650847 12531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "836c8d46b4524e61b21573ead0b4ce69"
I20260928 21:11:15.651054 12531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:15.652400 12531 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ced7e3ccdfa4f9984f09f74ed94821a in term 1.
I20260928 21:11:15.653205 12456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "34a0b04c5f144e9893c6466dcd1adc3a"
I20260928 21:11:15.649874 12488 master_runner.cc:186] Error getting master registration for 127.21.140.125:42595: OK, instance_id { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" instance_seqno: 1790629875344690 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:11:15.653419 12456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:15.653467 12488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:11:15.654778 12456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ced7e3ccdfa4f9984f09f74ed94821a in term 1.
I20260928 21:11:15.655575 12356 leader_election.cc:302] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ced7e3ccdfa4f9984f09f74ed94821a, 836c8d46b4524e61b21573ead0b4ce69; no voters:
I20260928 21:11:15.655896 12565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:15.656116 12565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [term 1 LEADER]: Becoming Leader. State: Replica: 2ced7e3ccdfa4f9984f09f74ed94821a, State: Running, Role: LEADER
I20260928 21:11:15.656445 12565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } }
I20260928 21:11:15.658213 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ced7e3ccdfa4f9984f09f74ed94821a. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.658330 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: This master's current role is: LEADER
W20260928 21:11:15.666244 12613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:15.666411 12613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:15.671921 12614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:15.672056 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.655026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.21.140.125:42595)
0928 21:11:15.655136 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.655141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.662295 (+ 7154us) server_negotiation.cc:407] Connection header received
0928 21:11:15.662550 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.662553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.662606 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.662683 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.663747 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.664563 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.665756 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.666154 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.666228 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.668140 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.668145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.668148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.668174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.669910 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.670671 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.670675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.670677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.670763 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.671090 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.671094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.671096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.671308 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.671405 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.671775 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.671895 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:11:15.673023 12614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:15.679776 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.655064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.21.140.124:36253)
0928 21:11:15.655178 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.655184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.662759 (+ 7575us) server_negotiation.cc:407] Connection header received
0928 21:11:15.662929 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.662933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.662983 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.663038 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.670119 (+ 7081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.670856 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.671946 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.672293 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.672370 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.675215 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.675219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.675222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.675250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.677212 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.677763 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.677766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.677768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.677867 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.678198 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.678201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.678203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.678742 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.679145 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.679481 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.679612 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:11:15.681751 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:15.655373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.21.140.126:36041)
0928 21:11:15.655945 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.655950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.672901 (+ 16951us) server_negotiation.cc:407] Connection header received
0928 21:11:15.673075 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.673078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.673128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.673179 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.674116 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.674876 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.675988 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:15.676320 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:15.676389 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:15.677952 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:15.677956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:15.677959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:15.678025 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:15.679735 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.680457 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.680461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.680462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.680555 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:15.680842 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:15.680846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:15.680847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:15.681046 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:15.681156 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:15.681480 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 21:11:15.681604 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496}
I20260928 21:11:15.685146 12456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [term 1 FOLLOWER]: Refusing update from remote peer 2ced7e3ccdfa4f9984f09f74ed94821a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:11:15.685180 12531 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [term 1 FOLLOWER]: Refusing update from remote peer 2ced7e3ccdfa4f9984f09f74ed94821a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:11:15.685880 12601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [LEADER]: Connected to new peer: Peer: permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:11:15.686141 12565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:11:15.689858 12580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ced7e3ccdfa4f9984f09f74ed94821a. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.690017 12580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:15.690366 12593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ced7e3ccdfa4f9984f09f74ed94821a. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.690501 12593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:15.704735 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.704896 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:15.706050 12593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.706176 12593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:15.706611 12601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.706722 12601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:15.707717 12580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ced7e3ccdfa4f9984f09f74ed94821a" member_type: VOTER last_known_addr { host: "127.21.140.126" port: 36041 } } peers { permanent_uuid: "34a0b04c5f144e9893c6466dcd1adc3a" member_type: VOTER last_known_addr { host: "127.21.140.125" port: 42595 } } peers { permanent_uuid: "836c8d46b4524e61b21573ead0b4ce69" member_type: VOTER last_known_addr { host: "127.21.140.124" port: 36253 } } }
I20260928 21:11:15.707840 12580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:11:15.710728 12616 mvcc.cc:205] Tried to move back new op lower bound from 7334419970798751744 to 7334419970692198400. Current Snapshot: MvccSnapshot[applied={T|T < 7334419970798751744}]
I20260928 21:11:15.711083 12618 mvcc.cc:205] Tried to move back new op lower bound from 7334419970798751744 to 7334419970692198400. Current Snapshot: MvccSnapshot[applied={T|T < 7334419970798751744}]
I20260928 21:11:15.711315 12617 mvcc.cc:205] Tried to move back new op lower bound from 7334419970798751744 to 7334419970692198400. Current Snapshot: MvccSnapshot[applied={T|T < 7334419970798751744}]
I20260928 21:11:15.713384 12614 catalog_manager.cc:1347] Generated new cluster ID: d3dc14bcaa8441de889e807d78a5735b
I20260928 21:11:15.713455 12614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:15.733359 12614 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:15.734709 12614 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:15.752717 12614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ced7e3ccdfa4f9984f09f74ed94821a: Generated new TSK 0
I20260928 21:11:15.753639 12614 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:11:15.782549 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.779249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.21.140.126:36041)
0928 21:11:15.779374 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.779379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.781340 (+ 1961us) server_negotiation.cc:407] Connection header received
0928 21:11:15.781542 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.781546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.781604 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.781659 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.782409 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59006: BlockingRecv error: recv got EOF from 127.0.0.1:59006 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 21:11:15.783999 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.780105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.21.140.125:42595)
0928 21:11:15.780213 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.780217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.783079 (+ 2862us) server_negotiation.cc:407] Connection header received
0928 21:11:15.783276 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.783279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.783330 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.783381 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.783881 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36632: BlockingRecv error: recv got EOF from 127.0.0.1:36632 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 21:11:15.785418 12563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.780732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.21.140.124:36253)
0928 21:11:15.780848 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.780852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.784441 (+ 3589us) server_negotiation.cc:407] Connection header received
0928 21:11:15.784695 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.784698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.784748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.784799 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.785295 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54200: BlockingRecv error: recv got EOF from 127.0.0.1:54200 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:11:15.789862 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.788350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.21.140.125:42595)
0928 21:11:15.788455 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.788459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.788905 (+ 446us) server_negotiation.cc:407] Connection header received
0928 21:11:15.789151 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.789154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.789227 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.789279 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.789765 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36634: BlockingRecv error: recv got EOF from 127.0.0.1:36634 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 21:11:15.794888 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.793363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.21.140.125:42595)
0928 21:11:15.793466 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.793470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.793998 (+ 528us) server_negotiation.cc:407] Connection header received
0928 21:11:15.794196 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.794199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.794245 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.794298 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.794786 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36644: BlockingRecv error: recv got EOF from 127.0.0.1:36644 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 21:11:15.800767 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.799223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.21.140.125:42595)
0928 21:11:15.799322 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.799326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.799830 (+ 504us) server_negotiation.cc:407] Connection header received
0928 21:11:15.800016 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.800019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.800090 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.800143 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.800665 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36654: BlockingRecv error: recv got EOF from 127.0.0.1:36654 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 21:11:15.808672 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.807076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.21.140.125:42595)
0928 21:11:15.807213 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.807217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.807730 (+ 513us) server_negotiation.cc:407] Connection header received
0928 21:11:15.807925 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.807928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.807978 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.808042 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.808566 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36668: BlockingRecv error: recv got EOF from 127.0.0.1:36668 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 21:11:15.821110 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.819324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.21.140.125:42595)
0928 21:11:15.819444 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.819449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.820017 (+ 568us) server_negotiation.cc:407] Connection header received
0928 21:11:15.820259 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.820263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.820325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.820380 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.820977 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36672: BlockingRecv error: recv got EOF from 127.0.0.1:36672 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 21:11:15.842852 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.840074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.21.140.125:42595)
0928 21:11:15.840202 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.840206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.840826 (+ 620us) server_negotiation.cc:407] Connection header received
0928 21:11:15.841066 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.841070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.841132 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.841219 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.842733 (+ 1514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36688: BlockingRecv error: recv got EOF from 127.0.0.1:36688 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 21:11:15.878690 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.877519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.21.140.125:42595)
0928 21:11:15.877658 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.877663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.877815 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:11:15.877927 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.877930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.878028 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.878150 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.878570 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36696: BlockingRecv error: recv got EOF from 127.0.0.1:36696 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 21:11:15.946237 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:15.944961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.21.140.125:42595)
0928 21:11:15.945131 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:15.945137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:15.945361 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:11:15.945428 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:15.945431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:15.945483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:15.945573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:15.946110 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36698: BlockingRecv error: recv got EOF from 127.0.0.1:36698 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 21:11:16.077629 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:16.076372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.21.140.125:42595)
0928 21:11:16.076604 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.076608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.076724 (+ 116us) server_negotiation.cc:407] Connection header received
0928 21:11:16.076907 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.076911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.076957 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.077062 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:16.077528 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36710: BlockingRecv error: recv got EOF from 127.0.0.1:36710 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260928 21:11:16.336939 12486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:11:16.335947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.21.140.125:42595)
0928 21:11:16.336108 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.336112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.336127 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:11:16.336182 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.336185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.336240 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.336337 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:16.336835 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36724: BlockingRecv error: recv got EOF from 127.0.0.1:36724 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 21:11:16.651873 12591 catalog_manager.cc:1259] Loaded cluster ID: d3dc14bcaa8441de889e807d78a5735b
I20260928 21:11:16.651965 12591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: loading cluster ID for follower catalog manager: success
I20260928 21:11:16.653728 12591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: acquiring CA information for follower catalog manager: success
I20260928 21:11:16.654974 12591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 836c8d46b4524e61b21573ead0b4ce69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:11:16.667774 12613 catalog_manager.cc:1259] Loaded cluster ID: d3dc14bcaa8441de889e807d78a5735b
I20260928 21:11:16.667860 12613 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: loading cluster ID for follower catalog manager: success
I20260928 21:11:16.669430 12613 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: acquiring CA information for follower catalog manager: success
I20260928 21:11:16.670555 12613 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34a0b04c5f144e9893c6466dcd1adc3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:11:16.855213 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:16.851229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.21.140.125:42595)
0928 21:11:16.851479 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.851482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.851537 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:11:16.851611 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.851613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.851738 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.851847 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:16.852107 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.852668 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.853578 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.853794 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.853874 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:16.854374 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:16.854820 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:16.854921 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:16.855029 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:11:16.855089 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260928 21:11:16.862155 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:16.858413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.21.140.124:36253)
0928 21:11:16.858734 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.858738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.858752 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:11:16.858843 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.858846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.858956 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.859073 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:16.859269 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.859791 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.860639 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.860855 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.860938 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:16.861244 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:16.861679 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:16.861773 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:16.861864 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:11:16.861911 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
I20260928 21:11:16.870215 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:16.866129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.21.140.124:36253)
0928 21:11:16.866244 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.866250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.866318 (+ 68us) server_negotiation.cc:407] Connection header received
0928 21:11:16.866529 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.866532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.866586 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.866659 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:16.866962 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.867756 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.868916 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.869248 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.869364 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:16.869642 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:16.869764 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:16.869817 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:16.870014 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:11:16.870073 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:11:16.870312 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:16.865503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.21.140.125:42595)
0928 21:11:16.865664 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.865667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.866145 (+ 478us) server_negotiation.cc:407] Connection header received
0928 21:11:16.866342 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.866346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.866404 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.866501 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:16.867150 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.867715 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.868961 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.869310 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.869403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:16.869827 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:16.869900 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:16.870011 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:16.870106 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 21:11:16.870158 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:11:16.871330 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:16.865671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.21.140.126:36041)
0928 21:11:16.866313 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:16.866318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:16.866333 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:11:16.866432 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:16.866438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:16.866604 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:16.866744 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:11:16.867100 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.867921 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.869140 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:16.869444 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:16.869562 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:16.869871 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:11:16.870262 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:11:16.870688 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:16.870754 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:11:16.871210 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":121,"threads_started":1}
I20260928 21:11:16.874760 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 12339
I20260928 21:11:16.881709 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 12414
I20260928 21:11:16.887698 22065 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczrM8B/build/debug/bin/kudu with pid 12488
2026-09-28T21:11:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3137 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6367 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (293411 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST